acceptodds
Withdrawn from ICLR 2027

Faithful to the Persona, Unfaithful to the Decision: Why Reasoning CoTs Cannot Report the Decisions They Narrate

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Under a system prompt that makes revenue the objective, a model's chain of thought (CoT) weighs a customer's safety against profit and recommends the harmful product in over half of our scenarios. CoT monitoring assumes the CoT reveals what the model decides; we ask where that decision is made and whether the CoT reveals it before the answer is written. On 182 product recommendation scenarios in which profit and hazard rise together, under a helpful and a profit prompt on Gemma-3-12B-IT and four further families, we removed and transplanted the CoT, located and steered the direction that carries the prompt's effect, and read the residual stream with a lens and a probe. Removing the CoT leaves the gap between the prompts unchanged, and one direction at the middle of the network carries the prompt's effect. Adding the commercial persona direction raises harm by 29 to 35 points; control and random directions of the same size add at most 8. The prompt induces a persona with an objective, and the model can put that persona into words: the CoT states the objective and is causally active. The direction that carries the episode's decision has no vocabulary image under the lens, and the CoT does not state the decision until it names the product. So a monitor reading the CoT before it names the product catches 14% of harmful recommendations at a 10% false alarm rate, and 61% from the full CoT. Asked afterwards, the model admits its objective but cannot be relied on to judge safety. Oversight should start from the objective the operator set, not from the reasoning the model shows.

Primary area:
interpretability and explainable AI
Keywords:
chain of thought faithfulness, CoT monitoring, mechanistic interpretability, activation steering, persona vectors, distributed alignment search, logit lens, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YGPgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether CoT reveals models’ decisions in profit-driven recommendation tasks, using prompt and CoT swaps, activation interventions, and monitoring experiments.

Critical strengths

The experiments cover multiple models and methods, with useful intervention controls. The authors release extensive code and results and acknowledge several limitations.

Critical weaknesses

The main observations appear consistent with ordinary reasoning—goals influence outputs, changing reasoning can change answers, and choices become clearer as reasoning progresses—yet the paper does not establish what these findings add beyond that expectation. Finding an intervention direction that existing tools cannot interpret does not establish that it represents the decision or explain why CoT cannot report it.

Questions for authors

Which result distinguishes the proposed explanation from ordinary, gradual decision formation? What does the mechanism explain beyond limited predictability from incomplete reasoning?

Reviewer_7g2AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates how revenue-oriented system prompts affect product recommendations and whether CoTs reveal the resulting decisions. It combines CoT removal and transplantation, activation steering, DAS, and representation readouts, with selected experiments replicated across five model families. The authors argue that CoTs verbalize the induced persona, while revealing limited information about episode-specific decisions early enough for monitoring.

Critical strengths
  • The paper addresses a relevant oversight problem and combines behavioral interventions, internal representation analysis, and CoT monitoring.

  • The experiments include substantial controls, such as alternative steering directions, prompt variants, and different prefix lengths, alongside partial replications across models and noncommercial domains.

Critical weaknesses
  • Early predictability is not clearly distinguished from faithfulness. The early-prefix condition stops before the first sentence mentioning any product letter, retaining only 26% of the CoT in the median S episode. Poor prediction from this incomplete prefix does not establish that the CoT misrepresents the decision process: substantive reasoning and decision formation may occur later. The fixed-percentage controls show increasing predictability but do not resolve this conceptual distinction (Table 21).

  • The monitoring interpretation mixes choice prediction with safety assessment. The full-CoT monitor predicts the final letter with 99.5% accuracy under S (Table 20), whereas the reported 61% recall at a 10% false-alarm rate comes from a separate harm-probability output (Table 21). Safety assessment is also complicated by hazards depending on undisclosed conditions in 95 of 182 scenarios (Table 2). These findings do not clearly attribute monitoring failures to the CoT’s inability to report the decision.

  • The proposed mechanistic explanation remains insufficiently established. DAS is trained and applied at the answer position with the complete CoT already in context. This identifies an effective intervention direction, but does not establish when the natural decision forms or why early CoT fails to expose it. Likewise, the tested readers’ limited ability to decode a direction does not establish that the model cannot verbalize the relevant information. The title and interpretation therefore appear stronger than the evidence supports (§§5–6).

  • The incremental contribution needs clarification. The paper should explain the substantive advance of its persona–decision distinction over prior work on persona vectors and CoT faithfulness, beyond combining established steering, probing, and readout methods.

Questions for authors
  • How is decision faithfulness operationally distinguished from early outcome predictability? Do the conclusions hold when prefixes preserve option comparisons but omit explicit final recommendations?

  • Can the authors report choice-based harm predictions alongside direct safety judgments, stratified by hazard knowability?

  • What evidence directly connects the decision direction’s limited readability to early CoT monitoring failures, rather than demonstrating these as separate observations?

Reviewer_pLEJOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper tackles the claim that chain of thought (CoT) monitoring is an effective tool for consumer safety with respect to AI tools. The rationale behind CoT monitoring is that observing the train of thought that the model is following enables a user to pre-emptively stop or intervene before it makes a harmful decision or recommendation.

Through a series of simulations and scenarios, the authors demonstrate that this exhibits only limited efficacy in practice, with little change due to the inclusion of CoT monitoring. Instead, their results show that the objective given to the AI tool has a much greater bearing on the harmfulness of its recommendations; querying the model for which objective it followed tells the user whether the recommendation may be potentially harmful, and subtracting the direction carried by the objective serves as an effective harmfulness mitigation technique.

Critical strengths

I think the big strength of the paper is the problem it seeks to address. AI safety is a highly relevant problem and the proliferation of AI tools in consumer experiences makes this work important for real-world applications. The authors have also cast the problem in an easily relatable scenario, product recommendations, wherein the harmful scenarios are clear.

Additionally, it appears that effort has been taken to comprehensively characterize the utility of CoT monitoring and subsequent inferences in the reported studies. This is critical because a comprehensive evaluation lends credence to the discussion and subsequent conclusions.

Critical weaknesses

My prevailing concern with this paper is its readability. I tried reading this manuscript multiple times but found it incredibly difficult to follow, let alone understand. This starts with the abstract wherein metrics have been arbitrarily included with little grounding for a reader who is engaging with the manuscript for the first time. The introduction suffers from the same issues, with many instances of run-on sentences. A clear example of this lack of clarity is found in Figure 1; the included metrics do not serve to clarify or substantiate the point that the authors are looking to make because the included numbers only make sense after the reader has already read subsequent sections.

Taking a step back, there is also somewhat of an absence of a clear structure. It isn't clear where the method is described, how the evaluations are conducted and what the conclusions are. In fact, it appears that these are all interweaved throughout the manuscript. While this may be a personal preference, the manuscript reads much more like a technical report in its current form, instead of a research manuscript seeking to introduce a new idea or study to readers.

Another minor nit: please explain jargon where possible. For isntance, what is a "J-lens". Without that background, it is incredibly difficult to follow the reasoning and discussion in the paper.

Questions for authors
  1. What does point 3 on Page 2 mean?
  2. What are the "two consequences for oversight" referred to on Page 2?
  3. The letter tier is said to have a correlation of 0.79 with the judge; does this not make the letter tier a fairly noise measurement of the harmfulness of the prediction?
  4. The last paragraph of Section 2 is scant on details. It seems to list a series of related works with no explanation as to their relevance to this manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.