acceptodds
Under review as a conference paper at ICLR 2027

The First Few Tokens Matter: Understanding Chain-of-Thought Controllability

Abstract

Chain-of-thought (CoT) monitoring is a promising way to oversee increasingly capable agents, in part because current models appear limited in their ability to control what they verbalize in their reasoning traces. We investigate the source of this constraint and link it to an entropy collapse in the first few tokens following the opening thinking tag. We exploit this mechanism to recover CoT controllability through two inference-time interventions on initial tokens: prefilling, where even a single token can substantially increase controllability, and banning frequent opening tokens combined with in-context examples, which improves average compliance by 34-41 percentage points across models. These interventions recover mainly addition and style controllability (adding or transforming information), while suppression controllability (avoiding or removing information) exhibits more varied responses, suggesting distinct underlying mechanisms. Consistent with this, the two groups behave as separate capabilities: across models, performance correlates positively within these groups, while cross-group correlations are mostly negative. Importantly, we find that associations with monitorability depend on the evaluation: addition and style controllability more consistently correlate positively with monitorability, while suppression sometimes shows negative associations. Aggregate controllability can therefore obscure these relationships, motivating separate reporting of control tasks and categories when evaluating CoT monitorability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Chain-of-Thought, Controllability, Monitorability, AI Safety, Evaluation Validity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_c9mDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the controllability of CoT reasoning in reasoning models and examines its implications for the effectiveness of model oversight. The authors find that current models’ difficulty in following instructions within their CoT is largely attributable to an initial-token bottleneck, where highly stereotyped opening tokens dominate the generation distribution and lead to an entropy collapse. Interventions such as prefilling compliant tokens or banning frequent opening tokens can substantially improve controllability, particularly for addition and style tasks. The study further suggests that suppression tasks may rely on mechanisms distinct from those underlying addition and style tasks, and that these different forms of controllability exhibit complex and heterogeneous relationships with monitorability. The authors therefore recommend reporting individual control tasks and categories separately in future AI safety evaluations, rather than relying solely on aggregate controllability measures that may obscure monitoring-relevant risks.

Critical strengths
  1. The paper identifies a concrete initiation bottleneck in CoT controllability and supports this hypothesis through targeted inference-time interventions. The observation that simple prefilling or first-token banning can substantially improve compliance suggests that low benchmark controllability may partially reflect a reasoning-initiation artifact rather than solely a lack of controllability.

  2. The paper provides useful evidence that current CoT controllability benchmarks may conflate distinct control capabilities. In particular, the separation between addition/style and suppression tasks, and their different relationships with monitorability, raises an important evaluation issue and motivates reporting these control types separately.

Critical weaknesses
  1. Gap between Safety Motivation and Empirical Setup (Proxy Evaluation Limitations): Both controllability and monitorability are evaluated through surface-level proxies (e.g., basic instruction-following and stylistic constraints) that remain distant from real safety-relevant monitor evasion. The current tasks do not capture whether a model possesses awareness of being monitored or strategically adapts to it, nor do they evaluate deliberate concealment, misleading reasoning, or evasion under adversarial incentives. Consequently, there is a clear disconnect between the paper's high-level safety motivation and its empirical setup, making it uncertain how well these findings generalize to real-world strategic evasion scenarios.

  2. The monitorability analysis is limited by weak discriminative power in several evaluations. Some monitorability benchmarks exhibit strong ceiling or floor effects, such as Memory Advice and GPQA Physics clustering near the ceiling and Scruples near the floor. This reduces meaningful variation across models and increases uncertainty in the reported controllability–monitorability correlations.

Questions for authors
  1. Lines 39–41: It would be helpful to provide relevant citations or concrete examples to support that this behavior has been observed in practice and represents a real concern.

  2. The motivation is somewhat indirect and difficult to follow because the central research gap is introduced only after several layers of background.

3 comments
AuthorsOpenReview ↗

Rebuttals (1/2)

We thank the reviewer for their detailed feedback!

We have made significant changes to our manuscript:

  • Revision of the writing, clarifying our major claims and improving flow. In particular, we clarify what conclusions can be drawn from the monitorability analysis and our research motivation.
  • Additional discussion of robustness and uncertainty of controllability and monitorability correlations

We address specific comments from the reviewer below.


“Gap between Safety Motivation and Empirical Setup (Proxy Evaluation Limitations): Both controllability and monitorability are evaluated through surface-level proxies (e.g., basic instruction-following and stylistic constraints) that remain distant from real safety-relevant monitor evasion. The current tasks do not capture whether a model possesses awareness of being monitored or strategically adapts to it, nor do they evaluate deliberate concealment, misleading reasoning, or evasion under adversarial incentives. Consequently, there is a clear disconnect between the paper's high-level safety motivation and its empirical setup, making it uncertain how well these findings generalize to real-world strategic evasion scenarios.”

We agree that the evaluated tasks do not directly measure strategic monitor evasion. The motivation for these proxies is that evaluating evasion across realistic incentives and settings is difficult; instruction-following tasks provide controlled, repeatable measurements of a potentially relevant capability. This rationale is explicit in the prior CoT-Control work, and such evaluations are also used in model safety assessments [1]. Our contribution is to investigate what their scores measure and what conclusions those scores support. In particular, sensitivity to default openings limits how directly low compliance can be interpreted as an inability to alter reasoning content. We have clarified (Section 4.2) that our experiments neither demonstrate strategic evasion nor validate these tasks as predictors of it.

“The monitorability analysis is limited by weak discriminative power in several evaluations. Some monitorability benchmarks exhibit strong ceiling or floor effects, such as Memory Advice and GPQA Physics clustering near the ceiling and Scruples near the floor. This reduces meaningful variation across models and increases uncertainty in the reported controllability–monitorability correlations.”

We agree that limited discrimination materially constrains the correlation analysis. We now propagate uncertainty from both controllability and monitorability measurements through a paired bootstrap with the twelve models held fixed. Forty-five of the 48 original controllability–monitorability intervals include zero, and baseline adjustment changes several suppression correlation signs. We therefore treat these associations as descriptive and do not infer a universal positive or negative relationship. Despite this uncertainty, the predominantly positive point estimates for addition and style tasks motivate closer examination of how these benchmarks relate. We agree that this pattern does not establish that the controllability proxies are invalid: our monitorability evaluations do not elicit evasion, and a model could be both more informative ordinarily and more capable of concealment when attempting it. We have clarified this distinction and identify a more direct validation question: whether controllability predicts the ability to reduce monitorability under an evasion objective. Our intervention results provide separate, stronger evidence that low scores on several controllability tasks can reflect difficulties with reasoning initiation.

“Lines 39–41: It would be helpful to provide relevant citations or concrete examples to support that this behavior has been observed in practice and represents a real concern.”

Thank you for this comment! We add a direct example from the GPT-6 Astra system card: in a controlled honeypot evaluation, making the model aware of CoT monitoring reduced CoT-only monitor recall [1]. While this is a controlled evaluation rather than evidence about its prevalence in deployment, these results highlight that investigating capability for this behavior represents a real concern.

AuthorsOpenReview ↗

Rebuttals (2/2)

“The motivation is somewhat indirect and difficult to follow because the central research gap is introduced only after several layers of background.”

Thank you for highlighting this lack of clarity! We have revised it to make the gap explicit immediately after introducing existing controllability results: why do models score poorly on these tasks, and what do these scores tell us about their ability to control reasoning content? These questions are connected: interpreting low scores as reassurance for monitoring depends on understanding what causes those failures. We now frame our interventions as investigating the source of low measured controllability, and our task comparisons and monitorability analyses as examining how these measurements should be interpreted.


We thank the reviewer for their detailed feedback which has helped significantly improve the clarity of our presentation of our research question and findings. We hope our responses have addressed the reviewer’s concern and welcome any further questions.


References: [1] OpenAI. GPT-6 Astra System Card. Technical report, OpenAI, September 2026. URL https://deploymentsafety.openai.com/gpt-6-astra/gpt-6-astra.pdf. Published September 3, 2026.

Thx for your clarification. I have raised my sore to 3, and hope the authors can incorporate this information into their final revision.

Reviewer_HSQgOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors show that reasoning models open almost every trace with "Okay" and that instructions barely move this. Also prefilling or banning the opener recovers much of the lost compliance. The authors then argue from twelve-model correlations that suppression is a distinct capability. The main takeaway appears to be that these benchmarks are under-elicited and conflate task types.

Critical strengths
  • The initiation result appears to be fairly actionable, i.e., instructions raise compliant openers' rank but leave them under 0.25 percent of mass (Fig. 11). Also one prefilled token or a two-token ban helps to restore compliance (Figs. 5 and 6). Can the authors clarify?
Critical weaknesses
  • My main issue is that there appears to be a confound. The above also explains suppression's "weaker response to compliant prefilling" (Sec. 3.4), i.e., a prefix that lacks a word doesn't demonstrate anything.
Questions for authors
  • Can the authors rerun Figs. 7 and 8 with instruction-minus-baseline suppression scores. Do the cluster and Fig. 8 signs survive?
  • Does a non-thinking or base Qwen3 show the same "Okay" collapse? If it's an RL artifact that one figure explains the mechanism
2 comments

We thank the reviewer for their detailed feedback!

We have made significant changes to our manuscript:

  • Revision of the writing, in particular we clarify what conclusions can be drawn from the observed differences between the two controllability groupings.
  • Additional discussion of robustness and uncertainty of controllability and monitorability correlations

We address specific comments from the reviewer below.

“The initiation result appears to be fairly actionable, i.e., instructions raise compliant openers' rank but leave them under 0.25 percent of mass (Fig. 11). Also one prefilled token or a two-token ban helps to restore compliance (Figs. 5 and 6). Can the authors clarify?”

Instructions affect the predicted distribution even when they do not change the sampled opening: compliant-token probability increases, but remains below 0.25% in the evaluated model-task pairs. Prefilling directly supplies an alternative opening, whereas banning excludes the dominant competitors and leaves the alternative to the model. Both can therefore improve compliance despite the limited effect of prompting alone. The actionable implication for evaluation is that instruction-only scores can understate elicitable performance on these proxy tasks; opening interventions provide a simple diagnostic of this sensitivity. We hope this explanation offers additional clarity.

“My main issue is that there appears to be a confound. The above also explains suppression's "weaker response to compliant prefilling" (Sec. 3.4), i.e., a prefix that lacks a word doesn't demonstrate anything.”

Thank you for raising this subtle confound! We agree that a prefix lacking a forbidden word conveys less information about the intended constraint than an uppercase or marker-prefixed example. We have added this qualification (in Section 3.3) and do not treat weaker prefill gains as evidence for distinct underlying mechanisms. Our capability distinction concerns the particular proxy tasks: across models, addition/style performance covaries positively, whereas suppression performance exhibits a different association pattern. This provides descriptive evidence for partially distinct task capabilities, without establishing a general separation between information addition and suppression or validating suppression tasks as measures of concealment.

“Can the authors rerun Figs. 7 and 8 with instruction-minus-baseline suppression scores. Do the cluster and Fig. 8 signs survive?”

Thank you for raising this important issue! We have repeated both analyses using instructed-minus-uninstructed compliance for the two suppression tasks. The broad control-task separation persists but weakens: the suppression-pair correlation changes from 0.74 to 0.69, and the mean cross-category correlation changes from −0.29 to −0.20, with eight of twelve adjusted cross-category correlations remaining negative.

The monitorability signs do not all survive. Both suppression tasks retain negative associations with GPQA and GPQA Physics, but first-person avoidance changes from −0.33 to +0.54 for WMDP, and both suppression correlations change from positive to negative for Scruples. We report these comparisons and bootstrap uncertainty in Appendix C. These changes qualify the individual monitorability associations, but preserve our broader conclusions: addition/style and suppression tasks exhibit different cross-model performance patterns, and neither scoring convention yields a consistent inverse relationship between controllability and monitorability. The sensitivity to baseline adjustment further emphasizes that interpreting these scores requires specifying what they measure. This supports our recommendation to report control tasks separately and validate their relationship to monitor evasion, rather than infer monitoring risk directly from aggregate controllability.

“Does a non-thinking or base Qwen3 show the same "Okay" collapse? If it's an RL artifact that one figure explains the mechanism”

We did not compare to base Qwen3, but we have done some analysis on Olmo checkpoints and included it in Appendix D. We found this opening entropy to mainly decrease in the thinking SFT and DPO stages. However, the base checkpoint of Olmo already shows ~80% of probability mass on “Okay” after inserting the begin think token. This probability mass on “Okay” and its leading space variant gets more concentrated throughout the post-training. Concentration therefore predates RL in this checkpoint sequence. However, the base model already encountered thinking traces during midtraining, and this comparison does not establish when the preference originated or which training data caused it.


We thank the reviewer for their detailed feedback which has helped significantly improve the clarity of our findings and the rigor in which we present them. We hope our responses have addressed the reviewer’s concern and welcome any further questions.

AuthorsOpenReview ↗

Summary of updates before discussion deadline

Dear Reviewer HSQg,

Thank you again for your feedback and suggested analyses. With the discussion period closing soon, we wanted to highlight three updates:

  1. Baseline-adjusted suppression: We repeated the correlation analyses using instruction-minus-baseline scores. The broad control-task separation persists, although several monitorability association signs change; we report and discuss both findings.
  2. Interpreting suppression: We explicitly acknowledge that compliant prefixes provide weaker demonstrations of suppression constraints and clarify that our capability distinction concerns the relationships of proxy tasks evaluated across models.
  3. Checkpoint comparison: We added an OLMo analysis showing that opening concentration is already present before RL and becomes substantially stronger throughout post-training.

We would appreciate your thoughts on whether these additions resolve your concerns. Please let us know if anything needs further clarification before discussion closes.

Reviewer_ReZkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines why reasoning models often fail to follow instructions in their chain-of-thought traces. It points to sharply concentrated distributions over opening tokens as a likely bottleneck, and finds that either prefilling compliant tokens or banning the most common openings raises compliance, primarily on addition and style tasks. Comparisons across models indicate that suppression behaves differently from other instruction types, and that controllability doesn't consistently relate to monitorability.

Critical strengths
  1. The paper identifies an interesting opening-token bottleneck and demonstrates substantial compliance improvements through simple inference-time interventions. One step forward towards the thinking chain hacking problem.
  2. It distinguishes suppression from addition and style control, providing a more nuanced analysis of the relationship between CoT controllability and monitorability.
Critical weaknesses
  1. Although several benchmarks are included, the actual question coverage is limited. The main suite contains 240 question–control pairs covering 178 distinct questions, with large portion of the pairs drawn from CommonsenseQA and ARC-Challenge. GPQA, MMLU-Pro, and HLE receive very limited coverage.
  2. The paper doesn't sufficiently examine whether improved instruction compliance comes at the cost of answer accuracy.
  3. The experiments show that changing the opening is effective, but do not sufficiently isolate the role of low entropy. The authors could compare compliant, noncompliant, and semantically neutral openings, or vary the concentration of the opening distribution while keeping its content as comparable as possible.
  4. Banning conventional openings may simply redistribute probability mass rather than establish low entropy as the cause of poor controllability.
  5. The paper doesn't sufficiently establish whether the bottleneck is specific to the beginning of reasoning or occurs more generally at format transitions.
Questions for authors

None

2 comments

We thank the reviewer for their detailed feedback!

We have made significant changes to our manuscript:

  • Revision of the writing, clarifying our major claims and improving flow. In particular, we clarify our use of low entropy to describe the bottleneck of a few dominant reasoning openings, rather than attributing low entropy as the cause of low CoT controllability.
  • Additional discussion of robustness and uncertainty of controllability and monitorability correlations

We address specific comments from the reviewer below.


“Although several benchmarks are included, the actual question coverage is limited. The main suite contains 240 question–control pairs covering 178 distinct questions, with large portion of the pairs drawn from CommonsenseQA and ARC-Challenge. GPQA, MMLU-Pro, and HLE receive very limited coverage.”

Although it is true we did not cover a large part of GPQA, MMLU-Pro, and HLE, we believe the coverage of questions we do have is sufficient for making the point that the opening bottleneck is a confound that partially decreases CoT control scores LLMs obtain on these common benchmarks, which is the main finding of this paper. Larger coverage of more difficult questions from GPQA etc would give a more complete picture of the controllability of particular models, but that is not what we’re trying to investigate in this submission. We thank you for raising this concern, as this was unclear in the text, so we added more explicit justification for this choice in the main text (Section 3.1). Particularly, we do not claim that the magnitude of the intervention effects is invariant to question difficulty.

“The paper doesn't sufficiently examine whether improved instruction compliance comes at the cost of answer accuracy.”

Thank you for raising this point! We have included an appendix figure (Figure 15) showing the accuracy changes of all interventions and discuss accuracy in the main text. For banning combined with ICL, the 95% bootstrap intervals are [−3.53, 1.10] percentage points for Qwen and [−0.33, 5.61] for OLMo. Some prefilling conditions retain accuracy decreases after answer-format recovery; these are reported in the appendix.

“The experiments show that changing the opening is effective, but do not sufficiently isolate the role of low entropy. The authors could compare compliant, noncompliant, and semantically neutral openings, or vary the concentration of the opening distribution while keeping its content as comparable as possible.” “Banning conventional openings may simply redistribute probability mass rather than establish low entropy as the cause of poor controllability.”

We agree with you that our submitted manuscript overstates the role of low entropy driving low controllability. We have edited our manuscript (primarily in the abstract, introduction, and discussion) to emphasize the role of a few common openings being highly concentrated as the mechanism driving low measured controllability on these common benchmarks. You are correct that our banning experiments redistribute probability mass and do not establish low entropy as the cause. In our writing, we clarify that we use low entropy to describe this “opening bottleneck” phenomenon that we uncover where probability mass is highly concentrated on a few default openings.

“The paper doesn't sufficiently establish whether the bottleneck is specific to the beginning of reasoning or occurs more generally at format transitions.”

Thank you for raising this question! Our central claim concerns a bottleneck at the beginning of reasoning, rather than its uniqueness to that position. Comparable effects at other format transitions would be compatible with our findings: the relevant result is that intervening at reasoning start substantially improves CoT instruction compliance. Establishing whether this reflects a broader phenomenon across format transitions is an interesting extension, but is not required for this conclusion.


We thank the reviewer for their detailed feedback which has helped significantly improve the clarity of our findings. We hope our responses have addressed the reviewer’s concern and welcome any further questions.

AuthorsOpenReview ↗

Summary of updates before discussion deadline

Dear Reviewer ReZk,

Thank you again for your feedback, which helped us clarify our claims and strengthen the analysis. With the discussion period closing soon, we wanted to briefly summarize the updates addressing your concerns:

  1. Answer accuracy: We added an appendix figure reporting accuracy changes across interventions, including bootstrap intervals and an analysis of answer formatting failures.
  2. Entropy and the opening bottleneck: We revised the manuscript to distinguish concentration on default openings from the stronger claim that low entropy itself causes poor controllability, and overall clarify the scope of our claims.

We would appreciate your thoughts on whether these updates address your concerns, and invite you to update your assessment if appropriate. We are happy to clarify any remaining issues before discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.