acceptodds
Under review as a conference paper at ICLR 2027

How Corrupted Chain-of-Thought Exemplar Answers Shift Internal Answer Preferences

Abstract

A correct generated answer does not reveal whether a model ignored an incorrect exemplar answer or recovered from its influence. This raises three questions: (i) does reasoning remove the internal effect of an incorrect exemplar answer, (ii) which components carry that effect, and (iii) does that effect track final-answer accuracy? We study multi-hop comparison questions from 2WikiMultiHopQA, replacing only each retrieved exemplar's answer line with either a plausible but incorrect competing answer from its own source question or an unrelated same-category answer, and adding a reasoning–answer factorial that separates the answer-line effect from exemplar reasoning that contradicts it. We measure the target question's gold-versus-foil logit margin before and after reasoning in Llama-3.1-8B-Instruct, with a same-item cue-probe replication in Qwen2.5-7B-Instruct. Reasoning–answer inconsistency is the largest resolved effect in both models, though weaker in Qwen, and in Llama reasoning preserves it mainly through trace-following. Causal interventions identify five layer-15 writer heads and distributed upper-layer attention as shared endpoints for inconsistency and the smaller answer-plausibility effect; a low-dimensional writer signal reaches the layer-16 MLP, and query direction mediates part of the aligned answer-plausibility effect. Neither the corruption-induced margin drop nor its trace-following component detectably predicts factorial generation failure, and two manipulations differing fourfold in margin effect produce similar accuracy reductions. These results separate internal answer competition from successful output recovery.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, in-context learning, chain-of-thought, retrieval-augmented generation, activation patching

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ebTSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work studies how corrupted chain of thought examples  affect a model’s internal answer preferences on multi-hop comparison questions. The paper is very unclear and I don’t think I was able to understand enough of the experimental setup or results in order to summarize it.

Critical strengths

Please see the weakness section.

Critical weaknesses

Unfortunately, despite a careful reading, I was unable to understand their experimental setup or results. Many terms or results are unclear or unexplained. The authors should improve their presentation so that I can properly evaluate the scientific contribution.

Questions for authors

N/A

Reviewer_gyLnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the effect of corrupted in-context examples on LMs' multi-hop question answering behavior. Performance across inputs with different corruptions is compared: the answers in in-context examples are manipulated, and the consistency of the in-context CoT and answer is manipulated. Additionally, the LMs' answers are compared with and without a CoT for the target question. Finally, the paper introduces interventions on single attention heads, investigating the how the inconsistency of the CoT and answers and the plausibility of in-context example answers are processed. The paper finds that the answering performance is particularly sensitive to the (in)consistency of the IC example's answer and reasoning, and identifies attention heads carrying that inconsistency and representations of plausibility of the example answers, I think.

Critical strengths

None

Critical weaknesses
  • The paper was extremely difficult to follow: new terminology / paraphrases were introduced all the time, and most methodological details were offloaded to the appendix, such that it was not possible for me to understand or evaluate the presented results and conclusions as it was unclear to me how they came about.
  • The conditions and the interventions are not very clearly introduced: e.g., I am not sure what exaclty the "reasoning-answer factorial" refers to (although it seems to be a central part of the findings), and how the reasoning for the target question was generated / used exactly.
Questions for authors

The following aspects are examples of what was not clear to me, and some suggest how this could be addressed:

  • example of an item in the different conditions might be helpful
  • a figure or table descriptively presenting the results (ie., absolute model performance in the different conditions) would help interpret the following results comparing conditions and interventions
  • what does "resolved" or "unresolved effect" mean?
  • could the authors please clarify how the heads discussed in section 5 were identified?
  • could the relation between the conditions shown on the y-axis in figure 2 and the "arms" presented in table 1 be clarified?
Reviewer_FKH7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates how corrupted or incorrect Chain-of-Thought (CoT) exemplars affect the internal answer preferences of Large Language Models (LLMs) in retrieval-augmented generation (RAG). It addresses a fundamental ambiguity: a correct final output does not prove that a model successfully ignored a bad exemplar, as it may have been internally influenced yet still managed to recover the right answer. Using multi-hop comparison questions from 2WikiMultiHopQA, the authors employ a reasoning-answer factorial framework and mechanistic interpretability tools on Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct to uncover how corruption signals are processed internally.

Critical strengths
  1. The authors cleanly isolates reasoning-answer inconsistency from simple answer-line content, proving that internal disruption is driven by semantic friction in the logic chain rather than mere "wrong answers".
  2. Moves beyond correlational readouts by using rigorous activation patching to pinpoint exact computational structures, such as layer-15 writer heads and upper-layer attention readers.
Critical weaknesses
  1. Internal logit margins completely fail to predict generation failures (AUC ~0.50), and manipulations with vastly different internal impacts yield identical accuracy drops.
  2. Almost all the deep causal stuff is tested only on Llama, while Qwen shows different and much divergent dynamics, meaning we don't really know if the results are general.
Questions for authors
  1. Given that internal margin drops and trace-following components completely fail to predict generation failures (AUCs near 0.50) and that a fourfold difference in internal margin effect results in nearly identical accuracy drops (~2 percentage points), how do the authors reconcile their internal mechanistic claims with actual task behavior? Specifically, if these internal logit shifts do not track output errors, what practical or behavioral phenomenon are the layer-15 writer circuits actually driving?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.