acceptodds
Under review as a conference paper at ICLR 2027

How LLMs Are Persuaded: A Few Attention Heads, Rerouted

Abstract

Large language models can answer a factual question correctly, yet switch to an incorrect answer when exposed to persuasive context. Why this happens inside the model remains unclear. In this work, we study this question through causal interventions in controlled multiple-choice tasks and identify that persuasion mainly acts through a small set of mid-layer attention heads, which we call *decision heads*. These heads do not compute the answer themselves. Instead, they copy the option they attend to, so changing their attention can directly change the model's answer. We find that this attention is controlled by a one-dimensional *option-routing feature* in the option-token representations. Persuasive text changes this feature, causing the decision heads to attend to the persuasion target instead of the correct option. We further trace the routing feature to shallower attention heads that read persuasive keywords from the input. Together, these results reveal a causal pathway: shallow heads read persuasive keywords and write the routing feature, which redirects the decision heads’ attention toward the persuasion target; the decision heads then copy the selected option, leading to the wrong answer. We verify each step through intervention and find the same mechanism across four open-source model families and in a source-selection task derived from Generative Engine Optimization (GEO). Finally, based on this mechanism, we introduce a low-rank update to the decision heads' key projections to reduce their sensitivity to persuasion.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, LLM persuasion, attention circuits, activation patching, generative engine optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WPmoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The problem that he paper is addressing is that the LLMs that answer a factual question correctly often switch to a wrong answer under persuasive context, and existing defenses suppress the symptom without locating the cause. This paper gives the first circuit-level mechanistic account of persuasion in multiple-choice QA through activation patching, the authors localize the effect to a sparse set of mid-layer attention heads -- "decision heads" (e.g., Llama-3 L17H24) -- that do not compute the answer but copy whichever option token they attend to. A band of shallower upstream heads constructs this feature by reading persuasive keywords from the input. The mechanism replicates across four model families (Llama-3, Qwen3, OLMo-2, Gemma-3) and a second task, source selection under GEO-style injection. Finally, a rank-2 update to the decision heads' key projections raises accuracy under persuasion by 12.3–41.3 points while preserving 95.2–96.4% of clean decisions — though reversing the update drops accuracy by 31–42 points.

Critical strengths
  1. First causal circuit for persuasion: a complete, intervention-validated chain (keywords upstream heads routing feature decision heads --> copied option), not just behavioral documentation of a well-known vulnerability.
  2. Strikingly compact and elegant: sparsity (one/few heads), tetrahedral choice geometry, and a 1D routing feature — the "vertex jump, not gradual erosion" framing genuinely changes how the phenomenon should be thought about.
  3. Strong causal methodology: restoration scores, attention-pattern-only patching, keyword-anonymization sufficiency/necessity tests, and negative controls (random heads, random corrections negligible).
  4. Replication across four model families and two tasks, including the practically relevant GEO source-selection setting — the mechanism is not a one-model curiosity.
  5. Mechanism defense: the rank-2 key-projection update is a direct --> payoff of the understanding, with large gains and high clean preservation, and the dual-use reversal is reported honestly.
Critical weaknesses
  1. Multiple-choice only: the "decision head copies an option" story does not directly extend to free-form generation, where persuasion also matters. The headline claim is about persuasion generally; the evidence is MC-specific.
  2. The routing feature rests on a rank-1 approximation of the QK circuit -- if the approximation is poor in some regimes, the clean 1D story may be oversimplified.
  3. The GEO setup is a fixed document with injected instruction, not an end- to-end retrieval-and-generation pipeline; the realistic attack has additional steps.
  4. Patching-based localization carries known artifact risks (Hydra/copy suppression); the mitigation via representation analysis is good but not airtight, and examples are filtered to persuasion successes, leaving resisted cases less characterized.
  5. The rank-2 defense is trained and tested on the same two benchmarks; generalization to new persuasion styles and multi-turn settings is untested.
  6. The relationship to correct-letter heads is relegated to an appendix; the main text should do more to establish what is new versus a relabeling of known heads.
Questions for authors
  1. Does the same circuit operate in free-form generation, or is there an analogue (e.g., a decision direction for open-ended answers)? Any pilot evidence beyond multiple-choice?
  2. How well does the rank-1 QK approximation reproduce full attention logits across models and tasks — and does the routing-feature story survive where the approximation is weakest? The main text should carry more of this than the appendix.
  3. On persuasion-resisted examples (model keeps the correct answer despite persuasive context), what does the routing feature do -- is resistance a failure of upstream writing, or a stronger clean signal?
  4. The GEO setup injects an instruction into a fixed retrieved document. Does the mechanism survive a realistic pipeline with retrieval ranking and multi-document synthesis?
  5. How does the rank-2 defense generalize to unseen persuasion styles (emotional appeals, multi-turn persuasion) and to model families beyond the four tested?
  6. Activation patching can induce compensatory effects (Hydra, copy suppression) that the authors themselves cite. What checks rule out that the decision-head sparsity is partly a patching artifact? 7.Upstream heads attend to persuasive keywords and adjacent punctuation. Is the keyword reading content-specific (e.g., "Nigeria") or does it respond to generic persuasion markers (certainty language, citations)? This decides whether the defense generalizes or is keyword- whack-a-mole.
4 comments
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analysis on generalization to free-form generation, persuasion resistance, realistic retrieval reranking, defense generalization, and the Hydra effect (Part 1/4)

(Question 1, Weakness 1) Does the same circuit operate in free-form generation, or is there an analogue (e.g., a decision direction for open-ended answers)? Any pilot evidence beyond multiple-choice?

Thank you for this suggestion. We tested all four models on 131 questions with nine persuasive appeals each, removing all answer choices and labels. Models independently generated short conversational answers to clean and persuasive prompts. When persuasion changed a correct answer to the incorrect target, our ablation jointly replaced the decision heads' outputs, including the auxiliary heads in Appendix R, with clean-run outputs. We compared the two answer spans token by token. The token immediately before their first difference is the answer decision position in each response. We patched from this position in the clean response to its counterpart in the persuasive response, then continued generation with the persuasive prompt and its own generated prefix.

We scored replies by matching the dataset's answer texts, ignoring capitalization and punctuation. Table 1 reports the percentage of persuasion-induced errors corrected by patching. Controls average recovery over five random head sets per model on the same cases, with each set matching the number of decision heads in each layer. Decision-head patching achieved higher recovery in every model, showing that these heads' causal role in persuasion-induced errors extends to short free-form answers without choices or labels.

Table 1. Persuasion-induced errors corrected by patching decision heads in free-form generation (%)

ModelDecision headsControl heads
Llama-3 8B40.5%6.4%
Gemma-3 12B23.8%1.8%
OLMo-2 7B35.5%1.4%
Qwen3 4B19.6%3.0%

(Question 2, Weakness 2) How well does the rank-1 QK approximation reproduce full attention logits across models and tasks — and does the routing-feature story survive where the approximation is weakest? The main text should carry more of this than the appendix.

We measure how well the rank‑1 QK approximation reproduces native attention logits using mean centered R2R^2 over all visible keys at the final query. R2=1R^2=1 means perfect reproduction. As shown in Table 2, fidelity is high across models and tasks, with scores ranging from 0.842 to 0.962. e.g., the approximation for OLMo 2’s L18H19 on NQ2 explains 96.2% of the centered logit variation on average per prompt.

Table 2. Rank‑1 QK fidelity to native attention logits (mean centered R2R^2).

Model / headNQ2GEO
Llama 3 8B / L17H240.9380.933
Gemma 3 12B / L29H110.9060.883
OLMo 2 7B / L18H190.9620.955
Qwen3 4B / L24H290.9240.842
Qwen3 4B / L24H310.9440.888

The routing feature retains causal influence over answer selection even where the approximation is weakest. We pooled cases across models and tasks, ranked them by centered logit error on the persuaded prompt, and evaluated the K worst-fitting cases for K = 20, 40, 60, 80, 100. For Qwen, we averaged the errors of the two decision heads. We injected the routing feature into the decision heads’ key projections at the clean option’s ending newline. The results are reported in Table 3. These routing features still restored the clean choice substantially, e.g. with 71.10% probability of restoration on average in the 60 worst-fitting cases. Both controls stayed below 6%.

Table 3. Expected clean-choice restoration under temperature‑1 sampling.

KLearned keyOrthogonal keyValue only
2051.10%2.00%2.40%
4066.00%4.60%5.00%
6071.10%4.20%4.50%
8068.80%3.70%4.00%
10070.40%3.70%4.00%
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analysis on generalization to free-form generation, persuasion resistance, realistic retrieval reranking, defense generalization, and the Hydra effect (Part 2/4)

(Question 3) On persuasion-resisted examples (model keeps the correct answer despite persuasive context), what does the routing feature do -- is resistance a failure of upstream writing, or a stronger clean signal?

We compared successful and resisted appeals on NQ2 questions that had both outcomes. Table 4 summarizes the projection of each option's representation onto the option routing feature, showing mean changes from clean (Δ) and the mean remaining margin: the correct option's projection minus the target option's projection under persuasion. Positive margins favor the correct option; negative margins favor the target. Across all four models, resistance is consistent with both weaker upstream writing, reflected in a smaller feature increase in the target option, and a stronger surviving clean signal, reflected in a smaller correct-option decrease. Upstream writing still occurs under resistance, but the correct option retains a small feature advantage. e.g., compared with successful persuasion, resisted appeals in Qwen3 4B show 0.564 less loss of the correct-option feature and 0.416 less gain in the target-option feature, leaving the correct option with an advantage of 0.290 over the target.

Table 4. Mean changes in routing-feature projections and remaining margins.

ModelOutcomeCorrect ΔTarget ΔCorrect − target
Llama 3 8BSucceeded−0.525+1.208−0.517
Llama 3 8BResisted−0.242+0.965+0.007
Gemma 3 12BSucceeded−0.839+1.130−0.435
Gemma 3 12BResisted−0.475+0.865+0.195
OLMo 2 7BSucceeded−0.319+0.595−0.258
OLMo 2 7BResisted−0.162+0.432+0.063
Qwen3 4BSucceeded−0.911+2.071−0.691
Qwen3 4BResisted−0.347+1.655+0.290

(Question 4, Weakness 3) The GEO setup injects an instruction into a fixed retrieved document. Does the mechanism survive a realistic pipeline with retrieval ranking and multi-document synthesis?

Realistic retrieval pipelines first retrieve candidate passages and then rerank them by relevance. We follow the BM25-plus-FIRST design evaluated in the FirstMistral study to test whether the GEO decision heads also influence passage selection. For each HotpotQA question, BM25 retrieves 100 passages from the benchmark's Wikipedia corpus. Our four study models rerank this full list one window of 20 consecutive passages at a time. Starting at positions 81–100, the window moves upward by 10 positions through the updated list after each reranking, until it reaches positions 1–20. Consecutive windows therefore share 10 passages, allowing passages promoted in one window to advance farther up the list in later steps. Within each window, passages are labeled A–T, and the model is asked to choose the most relevant one. Each label is a single token, so its next-token probability provides a score for that passage. Sorting these 20 scores determines the window's new order.

We attack the reranker with an instruction disguised as an editorial note in one passage, asking it to rank that passage first. We analyze cases where this promotes the passage from ranks 11–20 to first within a 20-passage group. At the final input token, we replace the previously identified GEO heads' outputs with those from the same input without the note. Recovery rate is the percentage of attack cases where patching restores the original first-ranked passage. Table 5 reports recovery counts and rates for the GEO heads, alongside the mean recovery rate of four random control head sets matched by layer and size. The decision heads produce substantially greater recovery, supporting their causal role in FIRST-style passage selection. e.g., OLMo restores the original first-ranked passage in 129 of 200 cases (64.5%), compared with 0.38% for the random control heads.

Table 5. Recovery rates for GEO decision heads and control heads in FIRST-style reranking.

ModelRecovered / attack-induced casesRecovery rate (GEO heads)Recovery rate (controls)
Llama-3 8B95 / 20047.5%0.25%
Gemma-3 12B77 / 20038.5%1.50%
OLMo-2 7B129 / 20064.5%0.38%
Qwen3 4B121 / 20060.5%0.38%
Combined422 / 80052.8%0.63%
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analysis on generalization to free-form generation, persuasion resistance, realistic retrieval reranking, defense generalization, and the Hydra effect (Part 3/4)

(Question 5, Weakness 5) How does the rank-2 defense generalize to unseen persuasion styles (emotional appeals, multi-turn persuasion) and to model families beyond the four tested?

We expanded our evaluation of rank-2 defense on the NQ2 question-answering task to six model families by adding LFM2.5 and Aya Expanse, covering models ranging from 350M to 32B parameters. For each model, we trained on two appeal types and tested on the withheld type, repeating this for credibility, emotion, and logic. We also evaluated corrections trained on all three single-turn appeal types on 100 multi-turn dialogues per model, without multi-turn training. In each dialogue, a simulated user presents three randomly sampled appeals of different persuasion styles, one per turn, continuing even if the model is persuaded before the final turn. Results are shown in Table 6. All entries report accuracy before → after correction, with multi-turn accuracy measured after the third appeal. The correction improves accuracy in every withheld-appeal setting and on multi-turn persuasion for all six models, supporting generalization across persuasion styles and model families. e.g., Llama’s accuracy on unseen emotional appeals increases from 58.7% to 75.4%, and its multi-turn accuracy increases from 49% to 58%.

Table 6. Generalization to withheld appeal types and multi-turn persuasion.

ModelCredibility withheldEmotion withheldLogic withheldMulti-turn accuracy
Llama-3 8B61.1% → 85.7%58.7% → 75.4%41.3% → 65.9%49% → 58%
Gemma-3 12B66.7% → 80.9%71.6% → 83.7%64.5% → 73.0%64% → 84%
OLMo-2 7B54.8% → 89.3%50.0% → 94.0%42.9% → 88.1%29% → 57%
Qwen3 4B51.6% → 73.0%55.6% → 78.6%50.0% → 69.8%49% → 56%
Aya Expanse 32B18.5% → 42.6%25.9% → 53.1%15.4% → 45.7%3% → 26%
LFM2.5 350M8.3% → 65.5%9.5% → 61.9%10.7% → 59.5%4% → 40%

(Question 6, Weakness 4) Activation patching can induce compensatory effects (Hydra, copy suppression) that the authors themselves cite. What checks rule out that the decision-head sparsity is partly a patching artifact?

We tested downstream compensation. We patched the decision head outputs from the clean run into the persuaded run at the final token and measured the direct effect by fixing all later attention head outputs and all MLP outputs from the patched layer onward to their unpatched persuaded values. We measured the total effect with the later heads and MLPs running normally, and the indirect effect is the total minus the direct effect. Table 7 reports mean changes in the clean-choice versus persuaded-choice logit margin. Each entry lists direct / indirect / total effects, and all values are positive. Indirect effects account for 76–95% of the mean total effect across all four models and both tasks. e.g., on Llama 3 8B with NQ2, the direct effect is +0.91 and the indirect effect is +3.84, yielding a total effect of +4.75. The net indirect effect reinforces the direct effect, which is the opposite sign relationship to Hydra-style compensation effects. These decision head interventions therefore produce net downstream amplification.

Table 7. Direct, indirect and total effects of decision-head patching.

ModelNQ2GEO
Llama 3 8B0.91 / 3.84 / 4.750.59 / 1.91 / 2.50
Gemma 3 12B0.54 / 10.14 / 10.680.20 / 3.17 / 3.37
OLMo 2 7B0.62 / 2.00 / 2.620.23 / 0.78 / 1.02
Qwen3 4B2.58 / 20.08 / 22.662.42 / 8.44 / 10.85
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analysis on generalization to free-form generation, persuasion resistance, realistic retrieval reranking, defense generalization, and the Hydra effect (Part 4/4)

(Question 7) Upstream heads attend to persuasive keywords and adjacent punctuation. Is the keyword reading content-specific (e.g., "Nigeria") or does it respond to generic persuasion markers (certainty language, citations)? This decides whether the defense generalizes or is keyword- whack-a-mole.

We analyzed the attention patterns of the identified upstream heads in each model and categorized their roles in Table 8. Entity readers attend to target names, values and references. Attribution readers attend to phrases introducing a source or authority, while negation readers attend to negative expressions. Attribute readers focus on the requested property or answer type, topic readers on subject matter, and modifier readers on descriptive or emphatic qualifiers. Format readers attend to punctuation, separators and boundary markers.

Table 8. Roles of the 8 identified upstream heads based on their attention patterns, according to where they attend to.

RoleLlamaGemmaOLMoQwen
Entity readerL13H18, L10H29, L11H12, L10H28, L10H2L11H0, L17H10, L19H0, L20H11L12H2, L11H29, L14H14, L11H27L17H4, L17H15, L17H27, L18H15, L19H27
Attribution reader——L13H14, L12H15—
Negation readerL9H15L19H11——
Attribute readerL9H17——L17H25
Topic readerL11H13L19H8L8H21L18H11, L18H21
Modifier reader—L20H2——
Format reader—L19H12L13H7—

To test whether heads assigned other roles also contribute causally, we compared patching only the entity readers with patching the complete set. In the corresponding persuasive prompts, we replaced the selected heads' outputs at all four answer options with their outputs from the anonymized prompts. Table 9 reports recovery of the average correct-answer probability: 0% means no change from the persuasive prompts, and 100% means reaching the same average correct-answer probability as the anonymized prompts. Patching all eight heads gives significantly greater recovery in all models, showing that the additional non-entity readers also contribute to the causal effect. e.g., in Gemma, entity-only patching achieves 40.09% recovery, compared with 82.38% for all eight heads.

Table 9. Correct-answer probability gap restored by entity readers or all eight upstream heads.

Patched headsLlamaGemmaOLMoQwen
Entity readers only24.90%40.09%24.56%31.69%
Complete head set45.57%82.38%54.74%53.59%
Reviewer_Td2YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper finds a circuit comprising a single attention head on several LLMs that leads the models to choose an incorrect answer in context over a correct answer known to the model. The authors claim that this attention head localizes persuasion.

Critical strengths
  1. The paper has good methodological rigor.
  2. The paper combines mechanistic analysis w representation analysis.
  3. The paper identifies a small circuit for the task in question, comprising a single head as far as I can tell!
Critical weaknesses

(A) Primarily, I am concerned with the framing of persuasion in the paper. It is unclear to me how the "persuasion" framing as defined in this paper is different from two things:

(1) Retrieving in-context vs. parametric information There is a large amount of existing literature that deals with retrieving information present in-context vs. information retrieved from parametric knowledge. The authors' definition of persuasion seems highly entangled with this retrieval from context vs. retrieval from parametric knowledge, since what they define as "persuasion" words are simply words where the factual information is different from the parametric information. Particularly the decision head, 17.24 routes rather than computes, suggesting it retrieves the right information from context/parametric memory, making this even more concerning.

References

  1. Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models (Jin et al., Findings 2024)
  2. Wu, Wenhao, et al. "Retrieval head mechanistically explains long-context factuality." International Conference on Learning Representations. Vol. 2025. 2025.(Wu et al, ICLR 2025)
  3. Kahardipraja, Patrick, et al. "The atlas of in-context learning: How attention heads shape in-context retrieval augmentation." Advances in Neural Information Processing Systems 38 (2026): 118164-118208.(Kahardipraja et al) 4.Characterizing Mechanisms for Factual Recall in Language Models (Yu et al., EMNLP 2023)
  4. From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models (Ma & Okazaki, Findings 2026)
  5. Retrieval Heads are Dynamic (Lin et al., ACL 2026)
  6. Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking (Zhang et al., EMNLP 2025)

(2) Sycophancy The credibility/emotion pieces and their influence on model outputs have been explored by other work, particularly in MCQA contexts.

References

  1. Chen, Sixing, et al. "Tracing mechanisms of sycophantic agreement in language models." arXiv preprint arXiv:2609.35822 (2026).
  2. Pandey, Manav. "LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit." arXiv preprint arXiv:2604.19117 (2026).
  3. Wang, Keyu, et al. "When truth is overridden: Uncovering the internal origins of sycophancy in large language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 39. 2026.
  4. Rimsky, Nina, et al. "Steering llama 2 via contrastive activation addition." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

At the very least, comparing the heads you identify with these baselines and underscoring what is different or similar about them as compared to other in-context learning/retrieval heads as well as sycophancy heads would be good.

(B) The MCQA setting is a tricky setting for doing mechanistic interpretability -- it's use is largely due to methodological convenience in interpretability. MCQA is an unreliable proxy for free-form ability [1,2,3]. Additionally, models are known to employ sub-mechanisms to encode the answer token, the index of the answer position, and the corresponding answer symbol (the mechanism largely identified as driving persuasion in the paper) when generating responses in retrieval-based MCQA tasks [4,5,6] -- have you investigated whether the persuasion mechanisms are not confounded with general MCQA mechanisms? What happens when these heads are ablated in a free-form setting?

References

  1. Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. Artifacts or abduction: How do LLMs answer multiple-choice questions without the question? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10308–10330, 2024.
  2. Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, and Jonas Geiping. Answer matching outperforms multiple choice for language model evaluation, 2025
  3. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. “my answer is C”: First-token probabilities do not match text answers in instruction-tuned language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 7407–7416, 2024.
  4. Mueller, Aaron, et al. "MIB: A mechanistic interpretability benchmark." arXiv preprint arXiv:2504.13151 (2025).
  5. Wiegreffe, Sarah, et al. "Answer, assemble, ace: Understanding how LMs answer multiple choice questions." International Conference on Learning Representations. Vol. 2025. 2025.
  6. Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy (Wiegreffe et al., EMNLP 2023)

(C) The papers used to cite persuasion are https://arxiv.org/abs/2510.04721, https://arxiv.org/abs/2502.08177, and https://aclanthology.org/2024.acl-long.858.pdf. Two of these are sycophancy papers and the last one defines a persuasive conversation as a multi-turn conversation with consistent misinformation and context-dependent information, rather than the single-turn conversation (as far as I can tell) setting that the results in this study are based on.

Questions for authors
  1. Is the circuit comprised on one head? Just 17.24 in Llama-3-8B, for eg? If not, do you have top-k analyses?
  2. Which dataset was used for patching, which dataset was used for eval? How did you prevent leakage here?
  3. How does persuasion based retrieval differ from in-context vs. parametric retrieval? How does it differ from sycophantic retrieval?
  4. What happens if you ablate your heads on a free-form persuasion retrieval task? How does this generalize?
  5. How is the persuasion circuit entangled with subspaces related to factuality, as well as subspaces related to other MCQA operations like answer pointer localizaton, answer choice localization as well as the answer itself?
  6. Have you done these investigations on true persuasive conversations like multi-turn ones?
22 comments
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analysis on head overlap with retrieval and sycophancy, entanglement with factuality and answer-choice subspaces, and generalization to free-form and multi-turn persuasion (Part 1/5)

(Question 1) Is the circuit comprised on one head? Just 17.24 in Llama-3-8B, for eg? If not, do you have top-k analyses?

No. In Llama-3-8B, 17.24 is the dominant downstream decision head. The circuit also includes upstream attention heads that help build the routing feature 17.24 uses to select an answer, as we discussed in detail in Section 4.3. For the top-k analysis, we show the top four NQ2 heads per model in Table 1. Each entry shows the head and the mean increase in correct-answer probability, when that head is patched individually. Across the models we studied, downstream answer selection is concentrated in one or two decision heads. In Qwen3-4B, we identified two such heads, 24.31 and 24.29. Patching them jointly increased correct-answer probability by 79.69% on NQ2.

Table 1. Top Four Attention Heads per Model by Individual Patching Effect on NQ2

ModelRank 1Rank 2Rank 3Rank 4
Llama-3-8B17.24 (+47.28 pp)30.27 (+12.00 pp)31.1 (+11.20 pp)15.11 (+9.22 pp)
Gemma-3-12B29.11 (+65.72 pp)28.6 (+12.46 pp)42.0 (+8.13 pp)45.3 (+7.63 pp)
OLMo-2-7B18.19 (+36.42 pp)18.18 (+11.42 pp)25.17 (+4.89 pp)21.23 (+4.76 pp)
Qwen3-4B24.31 (+36.10 pp)24.29 (+28.36 pp)35.25 (+21.86 pp)30.5 (+15.82 pp)

(Question 2) Which dataset was used for patching, which dataset was used for eval? How did you prevent leakage here?

We performed activation patching on NQ2 and GEO. We split each dataset 80:20 by question, keeping all NQ2 appeals and repeated GEO queries together. For each model and dataset, we ranked heads by mean recovery on the discovery set and measured the selected heads’ recovery on the held-out validation set. Results with this split are reported in Table 25, Appendix AC of the revised paper. For each model and dataset, we prevent leakage by ranking and selecting heads using only discovery-set recovery scores, then measure recovery for those fixed heads on the validation questions.

AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analysis on head overlap with retrieval and sycophancy, entanglement with factuality and answer-choice subspaces, and generalization to free-form and multi-turn persuasion (Part 2/5)

(Question 3, Weakness A) How does persuasion based retrieval differ from in-context vs. parametric retrieval? How does it differ from sycophantic retrieval?

Thank you for these references. We clarify the similarities and differences below.

Contextual versus parametric retrieval. Yu et al. study conflicts introduced through explicit factual statements and identify distinct heads with opposing contributions to memorized and contextual answer logits. Jin et al. identify memory heads and context heads that bring competing information to the final position. In our setting, logical, credibility-based, and emotional appeals change which option is selected by one or a few shared middle-layer decision heads. We identify the mechanism controlling this selection: upstream heads modify a routing feature at option-content positions, redirecting the decision heads’ attention from the correct option to the persuasion target. The same decision head can therefore transmit either answer, depending on its attention. Our analysis explains the answer switch through attention rerouting within a shared head, extending prior accounts of opposing answer contributions across distinct heads.

In the copying-based analyses of Wu et al. and Atlas, retrieval heads attend directly to the in-context occurrence of the next token being generated. Our decision heads exhibit a distinct attention pattern: rather than attending to that token itself, they attend to positions several tokens later in the input, typically near the end of the corresponding answer option. At these positions, the option identity is already encoded in the token representations, and the decision heads read and transmit this identity. Thus, while both mechanisms involve copying, retrieval heads copy tokens directly from context, whereas our decision heads select an option through its encoded representation. Ma and Okazaki build on copying-based retrieval-head identification, while dynamic-retrieval and QRHead studies extend retrieval to supporting evidence.

Sycophancy. Pandey identifies attention heads carrying a shared falsity signal across sycophancy and lying, even when the model ultimately agrees with a false statement; Wang et al. establish causal effects of late-layer representations; Rimsky et al. identify directions that steer sycophancy. Our work specifies the computational pathway connecting persuasive input, the option-side routing feature, decision-head attention, and the resulting answer selection. Regarding Chen et al., we thank the reviewer for bringing this work to our attention. We want to respectfully point out that this paper was first posted on arXiv on September 21, 2026, after the ICLR 2027 abstract deadline. We were unaware of it at the time of submission.

To evaluate how much our decision and upstream heads overlap with those identified in prior work, we apply Wu et al.’s and Pandey’s identification methods to the four models we study, identifying retrieval heads and shared sycophancy–lying heads, respectively. We compare these sets with our decision heads and upstream heads using Jaccard similarity: the number of shared heads divided by the total number of distinct heads across both sets. Each entry in Table 2 reports the Jaccard similarity between the head sets indicated by its row and column. Across models, Jaccard similarity with Wu et al.’s retrieval heads is uniformly near zero: 0% for decision heads and 1.64–3.28% for upstream heads. Pandey’s sycophancy–lying heads have zero overlap with either group in all four models.

Table 2. Head-level overlap (Jaccard similarity, %).

ModelOur headsRetrieval headsShared sycophancy–lying heads
Llama-3 8BDecision0.00%0.00%
Upstream2.56%0.00%
Gemma-3 12BDecision0.00%0.00%
Upstream2.00%0.00%
OLMo-2 7BDecision0.00%0.00%
Upstream1.64%0.00%
Qwen3 4BDecision0.00%0.00%
Upstream3.28%0.00%
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analysis on head overlap with retrieval and sycophancy, entanglement with factuality and answer-choice subspaces, and generalization to free-form and multi-turn persuasion (Part 3/5)

(Question 4, Weakness B) What happens if you ablate your heads on a free-form persuasion retrieval task? How does this generalize?

Thank you for this suggestion. We tested all four models on 131 questions with nine persuasive appeals each, removing all answer choices and labels. Models independently generated short conversational answers to clean and persuasive prompts. When persuasion changed a correct answer to the incorrect target, our ablation jointly replaced the decision heads' outputs, including the auxiliary heads in Appendix R, with clean-run outputs. We compared the two answer spans token by token. The token immediately before their first difference is the answer decision position in each response. We patched from this position in the clean response to its counterpart in the persuasive response, then continued generation with the persuasive prompt and its own generated prefix.

We scored replies by matching the dataset's answer texts, ignoring capitalization and punctuation. Table 3 reports the percentage of persuasion-induced errors corrected by patching. Controls average recovery over five random head sets per model on the same cases, with each set matching the number of decision heads in each layer. Decision-head patching achieved much higher recovery in every model, showing that these heads' causal role in persuasion-induced errors extends to short free-form answers without choices or labels.

Table 3. Persuasion-induced errors corrected by patching (%) in free-form persuasion tasks.

ModelDecision headsRandom heads
Llama-3 8B40.5%6.4%
Gemma-3 12B23.8%1.8%
OLMo-2 7B35.5%1.4%
Qwen3 4B19.6%3.0%
Reviewer_YGysOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how persuasive context causes LLMs to abandon correct answers in multiple-choice QA. Using activation patching on four open models and two tasks (FARM/NQ2 and a Geo-Bench source-selection task), the authors identify a few mid-layer “decision heads” that copy the attended option. They then extract a one-dimensional key-side “option-routing feature” from a rank-one QK approximation and trace it to shallower heads that attend to persuasive keywords. Finally, a rank-2 correction to the decision heads’ key projections improves accuracy under persuasion on NQ2 and reduces injected-source selection on GEO.

Critical strengths
  1. The causal analysis is careful and uses several complementary interventions: output patching, subspace-restricted patching, attention-pattern patching, feature steering with value-only and random-orthogonal controls, and replay/freezing mediation. Confidence intervals are reported throughout.
  2. The main localization replicates across four model families and two tasks. The appendices add useful robustness checks, including a RoPE ablation, target-name-free appeals, transfer to held-out appeal families, and BBQ.
  3. The defense follows directly from the mechanism. Its sign-reversal check (−BAᵀ lowers accuracy by 31–42 points) gives good evidence that the update acts on the identified pathway.
Critical weaknesses
  1. The circuit may not be specific to persuasion. Appendix Y shows that the same heads also mediate legitimate, evidence-driven answer changes on BBQ. This suggests the decision heads are a generic option-selection bottleneck for multiple-choice answering, consistent with prior work on MCQ circuits and on context-versus-memory heads, rather than a mechanism specific to persuasion.
  2. Persuasion is confounded with mentioning the target. The clean prompt removes any mention of the target. The upstream keyword-reading heads may therefore reflect simple entity salience rather than persuasion. A control in which the target is mentioned neutrally is needed.
  3. The upstream evidence is partial. Only about 11% of the selected heads’ attention falls on entity tokens. Joint patching recovers as little as 12% of the gap (Gemma). Persuasion also persists after target names are removed (Appendix Z). The claim of a “verified causal chain” is therefore overstated.
  4. Some geometric findings are expected by construction. Four well-separated clusters naturally form a 3D tetrahedron. The “sharp jump” is partly a product of the ≥0.10 margin filter, which removes intermediate cases. The “high-fidelity copy” claim also does not hold uniformly; for example, Gemma’s diagonal entries in Fig. 21 are only 0.63–0.69.
  5. The defense evaluation is incomplete. Because the same heads also handle legitimate use of context, the correction may suppress correct contextual updates (e.g., in RAG or knowledge-conflict settings), and this is not tested. Clean preservation on GEO is only 68–82%. The paper also lacks baselines such as prompting defenses, fine-tuning on synthetic data, and a parameter-matched LoRA.
  6. The scope is limited. The study uses only 4–12B models, small patching cohorts (e.g., 142 pairs from 40 questions for Gemma), and multiple-choice settings only.
Questions for authors

see above strengths and weaknesses

5 comments
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on persuasion specificity, neutral-mention controls, upstream causal mechanisms, geometric robustness, defense effectiveness and efficiency, and generalization (Part 1/5)

(Weakness 1) Persuasion Specificity

We would like to distinguish the persuasion circuit characterized in our paper from the decision heads that form one component of it. The decision heads are shared computational components. Their reuse during legitimate evidence-based updates on BBQ does not imply that the persuasion circuit itself is generic: distinct circuits can share computational components.

Our circuit-level account includes upstream processing of persuasive input, modification of the key-side option-routing feature, the resulting change in decision-head attention, and transmission of the selected option’s identity. Sections 4.2–4.3 and Appendix R provide causal evidence for these computations and their connections.

The upstream processing heads are more specific to persuasion, we compare how effectively interventions on these heads undo persuasion-induced and evidence-driven answer changes. On FARM, we transplant outputs of these heads from an entity-anonymized control into the corresponding persuasive run, testing restoration of the correct answer. On BBQ, we transplant outputs from the ambiguous condition into the informative run, testing whether the intervention reverses the evidence-supported answer back toward “cannot determine.” The results are reported in Table 1, which reports the percentage of the donor–recipient probability gap recovered: 0% indicates no movement toward the donor, and 100% indicates a full recovery. Across models, upstream processing heads show substantially greater recovery for persuasion-induced answer changes than for evidence-driven updates, supporting their specificity to persuasion. e.g., patching Gemma’s upstream heads recovers 82.4% of the gap on FARM but only 8.3% on BBQ.

Table 1: Gap recovery from upstream processing head interventions in persuasion-induced answer changes and evidence-based answer updating.

ModelFARMBBQ
Llama45.60%13.50%
Gemma82.40%8.30%
OLMo54.70%7.60%
Qwen53.60%25.40%
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on persuasion specificity, neutral-mention controls, upstream causal mechanisms, geometric robustness, defense effectiveness and efficiency, and generalization (Part 2/5)

(Weakness 2) Target-Entity Salience and Neutral-Mention Controls

Thank you for raising this concern. The majority of upstream keyword-reading heads attend to entities, while others attend to cues such as negation and modifiers. We constructed a control that mentions the persuasion target neutrally, without advocating for it, and made sure the model's answer remained correct. The results are reported in Table 2(a, b).

As for entity-reading heads, each entry in Table 2(a) shows how many times as much attention a target-entity token receives, on average, as another word token in the same persuasive appeal or neutral statement. We measure attention sent from the target option's answer text. The results show that these heads strongly attend to target entities in both contexts. e.g., in Llama, target-entity tokens receive 36.67 times as much attention per token as other words under persuasion, compared with 46.39 times under neutral mention.

Table 2(a): Entity reading heads' attention to target entities under persuasive and neutral mentions.

ContextLlamaGemmaOLMoQwen
Persuasive36.67×11.74×16.40×34.34×
Neutral46.39×13.54×13.97×61.61×

Although entity-reading heads show general entity salience, their outputs may still affect routing differently under persuasion. To show this, we jointly patched the entity-reading heads' outputs from persuasive prompts or neutral-mention controls into the same original prompt, without an added persuasive appeal or neutral statement. Table 2(b) reports the change in the projection of the target option's representation onto the routing feature relative to the unpatched prompt. Larger positive values indicate stronger routing toward the target. Across all four models, persuasive outputs produce larger projection increases than neutral outputs, indicating stronger routing effects under persuasion. e.g., in Llama, the projection increases by 0.370 when we patch persuasive outputs and by only 0.075 when we patch neutral outputs.

Table 2(b): Routing effects of entity-head outputs from persuasive and neutral contexts.

Output sourceLlamaGemmaOLMoQwen
Persuasive+0.370+0.181+0.334+0.293
Neutral+0.075+0.100−0.113+0.073

We also analyzed the attention patterns of the identified upstream heads in each model and categorized their roles in Table 3. Entity readers attend to target names, values and references. Attribution readers attend to phrases introducing a source or authority, while negation readers attend to negative expressions. Attribute readers focus on the requested property or answer type, topic readers on subject matter, and modifier readers on descriptive or emphatic qualifiers. Format readers attend to punctuation, separators and boundary markers.

Table 3: Roles of the 8 identified upstream heads based on their attention patterns, according to where they attend to.

RoleLlamaGemmaOLMoQwen
Entity readerL13H18, L10H29, L11H12, L10H28, L10H2L11H0, L17H10, L19H0, L20H11L12H2, L11H29, L14H14, L11H27L17H4, L17H15, L17H27, L18H15, L19H27
Attribution reader——L13H14, L12H15—
Negation readerL9H15L19H11——
Attribute readerL9H17——L17H25
Topic readerL11H13L19H8L8H21L18H11, L18H21
Modifier reader—L20H2——
Format reader—L19H12L13H7—

To test whether heads assigned other roles also contribute causally, we compared patching only the entity readers with patching the complete set. In the corresponding persuasive prompts, we replaced the selected heads' outputs at all four answer options with their outputs from the anonymized prompts. Table 4 reports recovery of the average correct-answer probability: 0% means no change from the persuasive prompts, and 100% means reaching the same average correct-answer probability as the anonymized prompts. Patching all eight heads gives significantly greater recovery in all models, showing that the additional non-entity readers also contribute to the causal effect. e.g., in Gemma, entity-only patching achieves 40.09% recovery, compared with 82.38% for all eight heads.

Table 4: Correct-answer probability gap restored by patching entity readers or all eight upstream keyword-reading heads.

Patched headsLlamaGemmaOLMoQwen
Entity readers only24.90%40.09%24.56%31.69%
Complete set45.57%82.38%54.74%53.59%
AuthorsOpenReview ↗

We sincerely thank you for your insightful comments — New experiments and analyses on persuasion specificity, neutral-mention controls, upstream causal mechanisms, geometric robustness, defense effectiveness and efficiency, and generalization (Part 3/5)

(Weakness 3) Partial Upstream Evidence

A moderate amount of attention to entity tokens does not necessarily imply a limited causal role. To assess their causal contribution, we ablated the information flow from entity tokens to option tokens. Table 5 reports the proportion of persuaded cases that revert to the correct answer when the contributions from entity tokens to option tokens are replaced with those from the corresponding anonymized prompt. This intervention substantially reduces persuasion success. e.g., in Llama, ablating entity-token contributions restores the correct answer in 81.6% of previously persuaded cases, demonstrating their strong causal involvement despite their moderate attention mass.

Table 5: Correct-answer restoration after patching entity contribution paths

ModelLlamaGemmaOLMoQwen
Correct answer restored81.60%60.00%61.00%69.60%

Refining upstream head selection strengthened the effect of joint patching. Table 6 reports the fraction of the donor–recipient gap in $P(\mathrm{correct})-P(\mathrm{target})$ closed by jointly patching the eight selected heads’ outputs at the answer-choice tokens. Recovery transfers outputs from the persuasive prompt into the anonymized prompt and measures the shift toward the persuasion target. Corruption transfers outputs from the anonymized prompt into the persuasive prompt and measures the shift toward the correct answer. Joint patching closes a substantial share of the probability-margin gap in both directions across all four models. e.g., in Gemma, these interventions close 43.0% and 82.4% of the gap, respectively.

Table 6: Probability gap closed by joint patching of selected upstream heads

DirectionLlamaGemmaOLMoQwen
Recovery72.50%43.00%60.50%40.80%
Corruption46.40%82.40%57.20%53.60%

In Appendix Z, we introduce a target-name-free appeal that replaces direct mentions of the persuasion target with indirect descriptions of the same entity. e.g., “Paris” is replaced with “the capital of France” in the appeal. Table 7 reports the attention allocated by entity-reading heads to these indirect mentions. Mention share measures the proportion of total attention from the target answer’s tokens directed toward indirect mentions, while final-token enrichment measures the ratio of mean attention per final token of an indirect mention to mean attention per other token in the persuasive passage. We find that these heads continue to attend to the persuasion target even when it is referenced indirectly. e.g., in Llama, indirect mentions receive 19.61% of the attention, and their final content tokens receive 9.65 times as much attention as other tokens in the persuasive passage.

Table 7: Attention to indirect target mentions on name-free appeals

MeasureLlamaGemmaOLMoQwen
Mention share19.61%12.93%16.09%21.52%
Final-token enrichment9.65×4.87×8.77×8.81×

(Weakness 4) Tetrahedral Geometry, Margin Filtering, and Copying Fidelity

For Gemma and Qwen, no switches were excluded on NQ2, and only one case was excluded for each model on GEO. To assess the filter’s effect, we restore excluded cases and quantify each jump using a normalized directional displacement score, J. This measures the head output’s movement toward the target vertex relative to the distance between the original and target vertices. A direct vertex-to-vertex shift gives $J$ = 1. The results are reported in Table 8, comparing the mean $J$ in the original margin-filtered cohort with the mean after restoring excluded non-tie pairs. “Restored pairs” counts the pairs added back; “after restoration” includes both the original and restored pairs. For Llama and OLMo, substantial target-directed displacement remains after restoration. These results indicate that the observed vertex-shift pattern is not solely produced by the margin filter. e.g., on Llama NQ2, restoring all 11 excluded pairs changes the mean $J$ only from 0.737 to 0.725.

Table 8: Effect of the Margin Filter on Normalized Jump Magnitude

DatasetModelRestored pairsMean J (original cohort)Mean J (after restoration)
NQ2Llama-3110.7370.725
NQ2OLMo-2860.8880.846
GEOLlama-3290.8520.841
GEOOLMo-21630.9620.841

We agree that Gemma’s lower diagonal cosines indicate imperfect vector copying. However, the transported representation still clearly identifies the corresponding output option. We quantify this using the cosine margin, defined as the similarity to the matching output vertex minus the largest similarity to any competing output vertex. For Gemma, this margin is 0.506 and 0.361 for NQ2 options C and D (the options corresponding to the highlighted entries in the question), respectively, supporting faithful transport of option identity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.