acceptodds
Under review as a conference paper at ICLR 2027

Decodable but Unresolved: Chain-of-Thought Conflicts in Reasoning Models

Abstract

A reasoning trace can support a correct answer to the wrong question. We study whether reasoning models preserve the user question when a supplied trace addresses another. Our diagnostic, CoT-Swap, first verifies that a model answers both questions correctly with matching traces, then swaps the trace and scores the output against both answers. Across factual and mathematical tasks, reasoning-tuned models frequently follow the trace's question, even when further reasoning is allowed. Yet mismatch remains linearly decodable on unseen question identities. Complementary training and state-intervention studies test what changes this behavior. Under matched training recipes, trace-token supervision increases redirection, while conflicting-prefix supervision attenuates it. Donor controls show that recovery depends more on sharing the user question than on sharing its answer alone. A low-rank map learned to predict matching-minus-mismatched state differences outperforms probe-direction steering in matched comparisons and partially restores user answers on unseen questions and new tasks, without test-time donors. The results distinguish answering ability, mismatch readout, and successful correction: neither demonstrated competence nor decodable conflict ensures that generation follows the user question.

Primary area:
interpretability and explainable AI
Keywords:
reasoning models, chain-of-thought, source choice, activation steering, mechanistic interpretability, robustness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rCVGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how reasoning models behave when the user question conflicts with a supplied chain-of-thought (CoT). The authors introduce CoT-Swap and show that reasoning-tuned models are more likely than instruct-tuned models to follow the injected CoT rather than the original question. Through controlled training, linear probing, and activation interventions, they examine the effects of trace supervision and mismatch decodability. Finally, a learned low-rank correction partially restores answers to the original question.

Critical strengths

The paper combines controlled training, linear probing, and activation interventions to investigate the phenomenon beyond behavioral observations. The finding that highly decodable mismatches can remain behaviorally unresolved is particularly interesting.

Critical weaknesses

W1. Limited practical relevance and unclear safety implications. The main experiments assume that a CoT from an unrelated question is inserted into the assistant's reasoning continuation, but it is unclear how often this situation would arise in realistic deployments. Table 2 shows that redirection largely disappears when the same CoT is presented as user-visible text. Moreover, STABLE is not necessarily a desirable outcome from a safety-monitoring perspective. Even when the final answer follows the user question, an unrelated internal CoT could undermine CoT-based monitoring. This distinction is not evaluated.

W2. Insufficient explanation of training effects. The approximately 46% increase in SWAPPED following trace-token supervision (Section 6) is surprising and lacks a clear explanation. While the authors separately average the answer and trace losses to avoid diluting the answer-token contribution, this differs from the standard practice of averaging the loss over all supervised tokens. Reporting results using the conventional whole-sequence mean loss would help establish whether the observed effect persists under a standard training objective.

W3. Potential confounding from the evaluation prompt. Appendix U.1 explicitly instructs all models to reason step by step inside <think> tags. This instruction may influence how models interpret and follow the injected CoT. An ablation removing this instruction while keeping the injected traces fixed would help establish whether the observed source-choice failures are amplified by the evaluation prompt.

W4. Limited qualitative analysis of continued reasoning. Although Section 5 shows that additional reasoning partially restores STABLE, the paper does not provide sufficient qualitative analysis of how models handle these conflicts. Examples distinguishing successful correction from cases where the model recognizes the conflict but still follows the injected CoT would help explain the gap between conflict detection and resolution.

W5. Insufficient details on human evaluation. Appendix B reports human annotation results but does not sufficiently describe annotator qualifications, recruitment, compensation, or detailed annotation guidelines. Given the 78.25% agreement between automatic matching and human adjudication, these details are important for assessing the reliability and reproducibility of the evaluation.

Questions for authors

See Weaknesses.

Reviewer_SFiNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces CoT-Swap, an evaluation framework that tests whether reasoning models answer the original user question or follow an injected chain-of-thought (CoT) from a different question, finding that reasoning-tuned models are substantially more susceptible to redirection than instruction-tuned models. Despite this, linear probes can reliably detect mismatches between the user question and the injected reasoning trace, suggesting that the relevant information remains represented internally but is not consistently used during generation. The authors show that conflicting-trace supervision reduces susceptibility and that learned low-rank activation corrections can partially restore correct answers, including on unseen questions and different tasks.

Critical strengths
  • The paper introduces a well-controlled evaluation framework that isolates reasoning redirection from ordinary factual or reasoning errors. By requiring models to answer both questions correctly before swapping their reasoning traces, CoT-Swap distinguishes answering the wrong question from simply lacking the knowledge needed to answer correctly
  • The authors combine behavioral evaluation with mechanistic interventions instead of relying exclusively on linear probing. Activation patching, comparisons between donors sharing the same question or answer, and learned state corrections provide additional evidence about which representations influence the model's final answer
Critical weaknesses
  • The evaluation used primarily uses relatively simple factual questions and constrained mathematical problems. It remains unclear whether the observed vulnerabilities generalize to complex multi-step reasoning, programming, long-context tasks, or conflicts between semantically related questions, where recognizing the intended task may be considerably more difficult.
Questions for authors

Can you evaluate the model in more complex settings, e.g. multi-step reasoning, and include an ablation on that?

Reviewer_jVeyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether reasoning models can preserve the original user question when provided with a conflicting chain-of-thought (CoT) trace. The authors introduce CoT-Swap, a diagnostic framework that first verifies a model's ability to answer two questions correctly and then replaces the reasoning trace associated with one question with that of another. Experiments across multiple models and benchmarks show that reasoning-tuned models are substantially more likely than instruct-tuned models to follow the conflicting trace.

The paper further investigates this phenomenon through controlled training experiments, linear probing, and activation interventions. The results suggest that CoT supervision increases susceptibility to redirection, while training with conflicting prefixes mitigates it. Although the mismatch remains linearly decodable from hidden states, this information is not necessarily used during generation. Finally, the authors propose a learned low-rank, donor-free correction that partially improves adherence to the user question, including on unseen question identities and across tasks.

Critical strengths
  1. A well-controlled framework for studying conflicts between user questions and reasoning traces.

CoT-Swap provides a clear way to distinguish failures of answering ability from failures to follow the intended question. The comparison between reasoning-tuned and instruct-tuned models is supported by multiple controls, including shared question sets, fixed trace lengths, human adjudication, and different trace-delivery conditions. These experiments make the central behavioral finding convincing and help rule out several alternative explanations.

  1. Comprehensive experimental investigation beyond behavioral observations.

The paper goes beyond documenting the phenomenon by investigating how training and internal representations affect source choice. In particular, the controlled training experiments help distinguish the effects of trace supervision and conflict supervision, while the probing and activation-intervention experiments demonstrate that decodable mismatch information does not automatically translate into correct behavior. The donor-free correction and its evaluation on held-out questions and additional tasks further strengthen the empirical contribution.

Critical weaknesses
  1. Limited practical generalizability of the primary experimental setting.

The main findings rely on injecting conflicting traces into assistant-side reasoning continuations, which requires a particular level of access to the model's execution context. As shown in Section 5 and Table 2, the same conflicting content produces substantially less redirection when presented through ordinary user-visible channels. This suggests that the observed vulnerability is highly dependent on the delivery mechanism rather than being a general consequence of exposure to misleading reasoning content.

Although the paper acknowledges this distinction, it remains unclear how frequently such assistant-side conflicts can arise in realistic deployments and under what circumstances they can be induced without privileged control over the reasoning continuation. This limitation matters because the practical significance of the reported vulnerability depends heavily on the accessibility of the assumed setting.

  1. Insufficient isolation and validation of the proposed correction method's advantages.

While the learned low-rank correction achieves promising improvements, the comparisons do not fully isolate which aspects of the proposed method account for its effectiveness. In Section 8.3, the learned rank-16 correction is compared primarily against rank-one probe and mean-difference directions. Consequently, the performance advantage may partly reflect the higher-dimensional intervention rather than the benefit of learning an input-dependent correction.

Furthermore, the human annotation study in Section 4.2 reports only 78.25% agreement between automatic and human-adjudicated outcome labels. Since the correction results are evaluated using automatic answer matching, it would be valuable to establish whether the reported improvements remain consistent under human evaluation, especially for the held-out generalization experiments.

These issues make it difficult to determine how much of the reported improvement can be attributed specifically to the proposed correction mechanism and how robust the measured gains are.

Questions for authors
  1. Regarding the practical applicability of CoT-Swap: Could the authors clarify which realistic deployment scenarios would allow conflicting traces to enter assistant-side reasoning continuations? In particular, can the authors provide evidence or concrete examples showing how the cache-reuse, tool-routing, or prior-turn-reuse conditions studied in Section 5 could arise in practice without direct control over the assistant's internal continuation?

  2. Regarding the effectiveness of the learned correction: Could the authors provide additional controlled comparisons that separate the effects of intervention rank and input dependence (e.g., a rank-one learned correction or a stronger rank-matched baseline)? Additionally, could the authors validate the baseline-to-correction improvements on a representative subset of held-out examples using human-adjudicated outcomes, given the discrepancy between automatic and human labels reported in Section 4.2?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.