acceptodds
Under review as a conference paper at ICLR 2027

Pressure Is Not Enough: A Causal Test of Whether Optimization Induces Encoded Chain-of-Thought

Abstract

If a chain-of-thought (CoT) monitor penalizes certain words, does optimizing against that penalty cause a model to develop an alternative, causally load-bearing representation that the monitor does not recognize? We test this on Coin Flip, a synthetic state-tracking task, using GRPO. We first separate two confounds. Across several discovery and adoption mechanisms, we find no reliable spontaneous non-literal encoding. Separately, an SFT-seeded Llama-3-8B-Instruct checkpoint demonstrates that the capability exists: bidirectional counterfactual interventions succeed in 42/42 trials before RL and 167/168 trials across four seeds after 150 GRPO steps. We then resume RL from a genuinely literal checkpoint and apply lexical pressure from the first step. The penalty fires on every sampled main rollout throughout 150 steps, yet no alternative representation emerges: final accuracy shows no statistically detectable difference from an unpenalized control (p=1.0), and all 160 final-checkpoint completions and 9,600 training rollouts remain literal. Thus, under the tested conditions, lexical pressure did not induce an alternative representation despite verified capability to use one when supplied. We additionally identify two apparent discoveries that fail stricter causal tests and find an initial semantic probe of monitor evasion to be inconclusive.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
steganographic reasoning, chain-of-thought, AI safety, reward hacking, reinforcement learning, language models, adversarial training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ydq4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims to study the emergence of steganographic or encoded reasoning in chain of thought using an stylized experiment from Skaf et al (the coin flip one). Basically, this is a state-tracking task, where the prompt gives a starting side (Heads or Tails) and then 3–8 instructions. Each instruction is either “same” (the coin stays in the same state) or “different” (it changes to the other state). They then do GRPO on a few small models, where, the model is penalized whenever its reasoning contains the literal state words (Heads, Tails, Head, Tail, H, T). They found negative results, and some false positives, which they detail in the paper.

Critical strengths
  • The paper picked a neat setting (coin flip) which they could then use to disambiguate similar-looking hypotheses about if the model is really stego or not. It is one among a few useful settings that has been used in other papers as well.
Critical weaknesses
  • It is a negative result on some small, undertrained (AFAICT, 9,600 rollouts across 8 runs is 1,200 per run, so w 8 samples per prompt, that works out to about one prompt per step, or roughly 150 prompts per run), and older models. Trivially, stronger models can control their CoT (as per Yueh-Han et al. 2026; https://arxiv.org/abs/2603.05706) so it seems more like there were errors in their selection of model or their training setup. It seems much more likely, that the culprit to which induces stego in CoT traces is not an overt lexical penalty but one related to thinking length, which is a more well-known problem. As such the research contribution is quite small.
  • The paper is very hard to read. It has basically no figures (the schematics are all also text!) and leaps into many engineering details that are irrelevant to the final scientific experiment or results. It also reports many things as if they are all of equal importance; it took quite awhile for the paper to get to the central experiment of interest.
Questions for authors

I do not have any questions whose answers could materially affect my assessment of the paper.

Reviewer_CfWbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether lexical penalties on visible chain-of-thought induce a non-literal representation that causally influences the answer. It distinguishes non-literal output (L1), causal use (L2), and concealment from a semantic reader (L3), using a synthetic Coin Flip task and LoRA-based GRPO training. Exploratory Qwen discovery/adoption experiments yielded no reliable encoding, although a prompt-format mismatch limits their interpretation. In the central experiment, a literal-CoT checkpoint is trained for 150 steps under BASELINE or MAIN rewards with four seeds each. All 9,600 training rollouts and 160 final evaluations remain literal, and both conditions obtain mean final accuracy 0.700. Separately, an SFT-seeded Llama-3-8B model with a redesigned code-token answer format achieves 42/42 successful counterfactual continuations before RL and 167/168 after RL. The authors also audit two apparent successes and appropriately treat their semantic-monitor probe as inconclusive. The evidence supports a limited failure-to-emerge result and a separate demonstration of taught causal code use.

Critical strengths
  1. Useful methodological discipline. The paper distinguishes visible token substitution from causal influence and concealment, checks counterfactual continuations rather than only output accuracy, and reports failures of initial interpretations and multi-seed replication. This is consequential because otherwise plausible-looking encodings and repeated checkpoints can easily be overinterpreted.
  2. Concrete behavioral evidence with appropriately limited claims. The central experiment starts from a verified literal policy, uses corrected chat formatting, and reports four seeds per condition and complete rollout classifications. The separate Llama experiment provides strong behavioral evidence that a taught code can influence subsequent state tracking and survive narration paraphrases. The authors explicitly avoid claiming semantic monitor evasion or a controlled PPO-versus-GRPO result.
Critical weaknesses
  1. Effective lexical pressure is not established. Appendix A.4 normalizes rewards within groups of eight. If the lexical penalty is constant within a group, subtracting it leaves both the normalized advantages and their clipped values unchanged. Thus 1,200/1,200 positive penalty events establish detector engagement, but not a gradient favoring an alternative representation. In addition, Table 4 shows that MAIN changes both positive bonuses as well as the lexical penalty relative to BASELINE, so the central comparison does not isolate lexical pressure.
  2. The capability control is not matched to the central negative test. The central initialization is inherited from the Qwen Direct RL branch, whereas successful L1+L2 is established with Llama-3-8B after changing tokenization and requiring the code token itself as the final answer (Section 4.7). The Qwen SFT results establish token use but do not establish the same successful L2 capability under the original answer format. Consequently, the claim that the capability confound has been removed for the central experiment is too strong. A matched model/task/answer-format capability demonstration, together with evidence that the corrected RL pipeline can favor a supplied valid alternative, would materially strengthen the causal interpretation. 3 . The manuscript needs substantial editorial and structural cleanup. Several inconsistencies make it harder to follow the experimental logic and, in some cases, leave the reader unsure how to interpret the evidence. The central experiment in Section 4.6 does not clearly restate the model and full configuration, so readers must infer them from earlier sections. The manuscript also contains unresolved or conflicting references: Appendix C has a “Figure ??” placeholder (PDF p. 13, line 701); Table 6 labels the main literal results as tier 1 even though the paper elsewhere classifies literal outputs as tier 2 (PDF p. 15, lines 770–772); and the account of the KL clamp and KL stopping rule is difficult to reconcile between Section 3.5 and Appendix E. In addition, a sentence in Section 4.4 is visibly broken (“33.3every completion answered Tails,” PDF p. 6, around line 300), the appendix’s displayed-equation numbering skips Eq. 8 while later text refers to equation numbers that are not clearly present, and Figure 6 appears to conflate a consistent state-to-code mapping with the separate constant-token bonus. The paper should also clarify whether the reported 0.384–0.396 penalty range is averaged by run or by time bin.
Questions for authors

larification / decisive additional evidence 1: Can the capability and pressure experiments be matched in model, task, output vocabulary, and final-answer format? Can PENALTY-ONLY be compared with BASELINE from that same literal checkpoint? These comparisons would directly affect my recommendation. Clarification 2: In Section 4.4, removing the decode-back sentence also removes a potentially valid mediator from code to answer. Why does this establish absence of L2, rather than dependence on decoding plus visible leakage? Clarification 3: Please reconcile the Table 6 tier-1/tier-2 contradiction and distinguish the KL clamp from the KL stopping rule in Section 3.5 and Appendix E. Please also specify whether the 0.384-0.396 penalty range in Section 4.6 is averaged per run or per time bin, and whether it includes the annealed coefficient.

Reviewer_ebMUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate whether GRPO based optimization that penalizes explicit use of terms like “Heads” or “Tails” causes models to employ alternative encodings. They follow the experimental and Coin Flip task setup of Skaf et al (2025), and determine that models are able to learn “non-literal representations” from only lexical pressure. They validate that these results hold in a threefold hierarchy of claims, finding that the first two elements in the hierarchy - use of non-literal representations and causal load-bearingness as in Lanham et al (2023) - are both respected, but not necessarily the third, namely monitor evasion as discussed in Skaf et al (2025) (and the unattributed Zolkowski et al (2025)).

Critical strengths
  • The authors produce a limited but positive result showing that GRPO can be used to achieve non-literal representations in 8B scale models with small amounts of training.
  • They further verify that this learning encoding is causally load-bearing to completing the task, even after paraphrasing the CoT. This is an important validation to ensure that the learned mapping is not merely vestigial to the CoT.
Critical weaknesses
  • The reviewer found the paper rather hard to read and structured awkwardly. For example, the use of the word “literal” is key to the results, but is used without definition or explanation until Figure 1 in Section 3. Similarly, “p_CoT” is used 3 times before finally being defined in Appendix A.2. The authors refer in Section 4.3 to “with the untrained Yelt/Yark”, but this doesn’t appear anywhere else and it is not clear what it is referring to.
  • The authors maintain that “decode back” is required to have state tracking accuracy reported correctly in the answer section, but don’t attempt simple resolutions. For example, they could employ a light SFT phase after the last RL step to ensure the model reports back in Tails/Head in the answer section. If there is reason not to try such attempts, this should be reported in section 4.
  • This is a fairly narrow and limited contribution that essentially tests reproduction of one result of Skaf et al (2025) with GRPO instead of PPO. It is hard to claim novelty without more exploration of model families/sizes, task types, and encoding methods beyond direct literal-token to non-literal-token substitution.
Questions for authors
  • The results presented in Section 4 are a bit confusing, ewspecciallyt . For example, can you clarify whether the paraphrasing methodology uses the trained model to paraphrase itself, or some other model, and if so which and using what template? What are the values of the two cut points?
  • Can the authors report how the learned representation is acquired during training? For example, does it gradually or suddenly transition from using “Heads/Trails” to “Nomo/Nib” or some other coding? There might be a visual way to show this with a training chart.
  • In the hierarchy of claims, L3 is dependent on the kind of monitor. The reviewer was surprised not to see a clear set of monitor ablations, such as that similar to Appendix B.3.1 of “Early Signs of Steganographic Capabilities in Frontier LLMs” of Zolkowski et al (2025). Can clarify what kind of monitors were used and tested, as referred to in the first sentence of the abstract and throughout the paper? Is this the same as the “judge model probe”, and if so, what model and parameters were used for this monitor?
  • In Sec 4.2, re: "corrupting an intermediate state changed the final answer in 47.5-75.0% of cases". Can the authors verify that the problems tested have consistent answers across rollouts to verify that answer change is due to the corruption and not due to a stochastic answer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.