- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The problem that he paper is addressing is that the LLMs that answer a factual question correctly often switch to a wrong answer under persuasive context, and existing defenses suppress the symptom without locating the cause. This paper gives the first circuit-level mechanistic account of persuasion in multiple-choice QA through activation patching, the authors localize the effect to a sparse set of mid-layer attention heads -- "decision heads" (e.g., Llama-3 L17H24) -- that do not compute the answer but copy whichever option token they attend to. A band of shallower upstream heads constructs this feature by reading persuasive keywords from the input. The mechanism replicates across four model families (Llama-3, Qwen3, OLMo-2, Gemma-3) and a second task, source selection under GEO-style injection. Finally, a rank-2 update to the decision heads' key projections raises accuracy under persuasion by 12.3–41.3 points while preserving 95.2–96.4% of clean decisions — though reversing the update drops accuracy by 31–42 points.
Critical strengths
- First causal circuit for persuasion: a complete, intervention-validated chain (keywords upstream heads routing feature decision heads --> copied option), not just behavioral documentation of a well-known vulnerability.
- Strikingly compact and elegant: sparsity (one/few heads), tetrahedral choice geometry, and a 1D routing feature — the "vertex jump, not gradual erosion" framing genuinely changes how the phenomenon should be thought about.
- Strong causal methodology: restoration scores, attention-pattern-only patching, keyword-anonymization sufficiency/necessity tests, and negative controls (random heads, random corrections negligible).
- Replication across four model families and two tasks, including the practically relevant GEO source-selection setting — the mechanism is not a one-model curiosity.
- Mechanism defense: the rank-2 key-projection update is a direct --> payoff of the understanding, with large gains and high clean preservation, and the dual-use reversal is reported honestly.
Critical weaknesses
- Multiple-choice only: the "decision head copies an option" story does not directly extend to free-form generation, where persuasion also matters. The headline claim is about persuasion generally; the evidence is MC-specific.
- The routing feature rests on a rank-1 approximation of the QK circuit -- if the approximation is poor in some regimes, the clean 1D story may be oversimplified.
- The GEO setup is a fixed document with injected instruction, not an end- to-end retrieval-and-generation pipeline; the realistic attack has additional steps.
- Patching-based localization carries known artifact risks (Hydra/copy suppression); the mitigation via representation analysis is good but not airtight, and examples are filtered to persuasion successes, leaving resisted cases less characterized.
- The rank-2 defense is trained and tested on the same two benchmarks; generalization to new persuasion styles and multi-turn settings is untested.
- The relationship to correct-letter heads is relegated to an appendix; the main text should do more to establish what is new versus a relabeling of known heads.
Questions for authors
- Does the same circuit operate in free-form generation, or is there an analogue (e.g., a decision direction for open-ended answers)? Any pilot evidence beyond multiple-choice?
- How well does the rank-1 QK approximation reproduce full attention logits across models and tasks — and does the routing-feature story survive where the approximation is weakest? The main text should carry more of this than the appendix.
- On persuasion-resisted examples (model keeps the correct answer despite persuasive context), what does the routing feature do -- is resistance a failure of upstream writing, or a stronger clean signal?
- The GEO setup injects an instruction into a fixed retrieved document. Does the mechanism survive a realistic pipeline with retrieval ranking and multi-document synthesis?
- How does the rank-2 defense generalize to unseen persuasion styles (emotional appeals, multi-turn persuasion) and to model families beyond the four tested?
- Activation patching can induce compensatory effects (Hydra, copy suppression) that the authors themselves cite. What checks rule out that the decision-head sparsity is partly a patching artifact? 7.Upstream heads attend to persuasive keywords and adjacent punctuation. Is the keyword reading content-specific (e.g., "Nigeria") or does it respond to generic persuasion markers (certainty language, citations)? This decides whether the defense generalizes or is keyword- whack-a-mole.
4 comments
We sincerely thank you for your insightful comments — New experiments and analysis on generalization to free-form generation, persuasion resistance, realistic retrieval reranking, defense generalization, and the Hydra effect (Part 2/4)
(Question 3)
On persuasion-resisted examples (model keeps the correct answer despite persuasive context), what does the routing feature do -- is resistance a failure of upstream writing, or a stronger clean signal?
We compared successful and resisted appeals on NQ2 questions that had both outcomes. Table 4 summarizes the projection of each option's representation onto the option routing feature, showing mean changes from clean (Δ) and the mean remaining margin: the correct option's projection minus the target option's projection under persuasion. Positive margins favor the correct option; negative margins favor the target. Across all four models, resistance is consistent with both weaker upstream writing, reflected in a smaller feature increase in the target option, and a stronger surviving clean signal, reflected in a smaller correct-option decrease. Upstream writing still occurs under resistance, but the correct option retains a small feature advantage. e.g., compared with successful persuasion, resisted appeals in Qwen3 4B show 0.564 less loss of the correct-option feature and 0.416 less gain in the target-option feature, leaving the correct option with an advantage of 0.290 over the target.
Table 4. Mean changes in routing-feature projections and remaining margins.
| Model | Outcome | Correct Δ | Target Δ | Correct − target |
|---|---|---|---|---|
| Llama 3 8B | Succeeded | −0.525 | +1.208 | −0.517 |
| Llama 3 8B | Resisted | −0.242 | +0.965 | +0.007 |
| Gemma 3 12B | Succeeded | −0.839 | +1.130 | −0.435 |
| Gemma 3 12B | Resisted | −0.475 | +0.865 | +0.195 |
| OLMo 2 7B | Succeeded | −0.319 | +0.595 | −0.258 |
| OLMo 2 7B | Resisted | −0.162 | +0.432 | +0.063 |
| Qwen3 4B | Succeeded | −0.911 | +2.071 | −0.691 |
| Qwen3 4B | Resisted | −0.347 | +1.655 | +0.290 |
(Question 4, Weakness 3)
The GEO setup injects an instruction into a fixed retrieved document. Does the mechanism survive a realistic pipeline with retrieval ranking and multi-document synthesis?
Realistic retrieval pipelines first retrieve candidate passages and then rerank them by relevance. We follow the BM25-plus-FIRST design evaluated in the FirstMistral study to test whether the GEO decision heads also influence passage selection. For each HotpotQA question, BM25 retrieves 100 passages from the benchmark's Wikipedia corpus. Our four study models rerank this full list one window of 20 consecutive passages at a time. Starting at positions 81–100, the window moves upward by 10 positions through the updated list after each reranking, until it reaches positions 1–20. Consecutive windows therefore share 10 passages, allowing passages promoted in one window to advance farther up the list in later steps. Within each window, passages are labeled A–T, and the model is asked to choose the most relevant one. Each label is a single token, so its next-token probability provides a score for that passage. Sorting these 20 scores determines the window's new order.
We attack the reranker with an instruction disguised as an editorial note in one passage, asking it to rank that passage first. We analyze cases where this promotes the passage from ranks 11–20 to first within a 20-passage group. At the final input token, we replace the previously identified GEO heads' outputs with those from the same input without the note. Recovery rate is the percentage of attack cases where patching restores the original first-ranked passage. Table 5 reports recovery counts and rates for the GEO heads, alongside the mean recovery rate of four random control head sets matched by layer and size. The decision heads produce substantially greater recovery, supporting their causal role in FIRST-style passage selection. e.g., OLMo restores the original first-ranked passage in 129 of 200 cases (64.5%), compared with 0.38% for the random control heads.
Table 5. Recovery rates for GEO decision heads and control heads in FIRST-style reranking.
| Model | Recovered / attack-induced cases | Recovery rate (GEO heads) | Recovery rate (controls) |
|---|---|---|---|
| Llama-3 8B | 95 / 200 | 47.5% | 0.25% |
| Gemma-3 12B | 77 / 200 | 38.5% | 1.50% |
| OLMo-2 7B | 129 / 200 | 64.5% | 0.38% |
| Qwen3 4B | 121 / 200 | 60.5% | 0.38% |
| Combined | 422 / 800 | 52.8% | 0.63% |
We sincerely thank you for your insightful comments — New experiments and analysis on generalization to free-form generation, persuasion resistance, realistic retrieval reranking, defense generalization, and the Hydra effect (Part 3/4)
(Question 5, Weakness 5)
How does the rank-2 defense generalize to unseen persuasion styles (emotional appeals, multi-turn persuasion) and to model families beyond the four tested?
We expanded our evaluation of rank-2 defense on the NQ2 question-answering task to six model families by adding LFM2.5 and Aya Expanse, covering models ranging from 350M to 32B parameters. For each model, we trained on two appeal types and tested on the withheld type, repeating this for credibility, emotion, and logic. We also evaluated corrections trained on all three single-turn appeal types on 100 multi-turn dialogues per model, without multi-turn training. In each dialogue, a simulated user presents three randomly sampled appeals of different persuasion styles, one per turn, continuing even if the model is persuaded before the final turn. Results are shown in Table 6. All entries report accuracy before → after correction, with multi-turn accuracy measured after the third appeal. The correction improves accuracy in every withheld-appeal setting and on multi-turn persuasion for all six models, supporting generalization across persuasion styles and model families. e.g., Llama’s accuracy on unseen emotional appeals increases from 58.7% to 75.4%, and its multi-turn accuracy increases from 49% to 58%.
Table 6. Generalization to withheld appeal types and multi-turn persuasion.
| Model | Credibility withheld | Emotion withheld | Logic withheld | Multi-turn accuracy |
|---|---|---|---|---|
| Llama-3 8B | 61.1% → 85.7% | 58.7% → 75.4% | 41.3% → 65.9% | 49% → 58% |
| Gemma-3 12B | 66.7% → 80.9% | 71.6% → 83.7% | 64.5% → 73.0% | 64% → 84% |
| OLMo-2 7B | 54.8% → 89.3% | 50.0% → 94.0% | 42.9% → 88.1% | 29% → 57% |
| Qwen3 4B | 51.6% → 73.0% | 55.6% → 78.6% | 50.0% → 69.8% | 49% → 56% |
| Aya Expanse 32B | 18.5% → 42.6% | 25.9% → 53.1% | 15.4% → 45.7% | 3% → 26% |
| LFM2.5 350M | 8.3% → 65.5% | 9.5% → 61.9% | 10.7% → 59.5% | 4% → 40% |
(Question 6, Weakness 4)
Activation patching can induce compensatory effects (Hydra, copy suppression) that the authors themselves cite. What checks rule out that the decision-head sparsity is partly a patching artifact?
We tested downstream compensation. We patched the decision head outputs from the clean run into the persuaded run at the final token and measured the direct effect by fixing all later attention head outputs and all MLP outputs from the patched layer onward to their unpatched persuaded values. We measured the total effect with the later heads and MLPs running normally, and the indirect effect is the total minus the direct effect. Table 7 reports mean changes in the clean-choice versus persuaded-choice logit margin. Each entry lists direct / indirect / total effects, and all values are positive. Indirect effects account for 76–95% of the mean total effect across all four models and both tasks. e.g., on Llama 3 8B with NQ2, the direct effect is +0.91 and the indirect effect is +3.84, yielding a total effect of +4.75. The net indirect effect reinforces the direct effect, which is the opposite sign relationship to Hydra-style compensation effects. These decision head interventions therefore produce net downstream amplification.
Table 7. Direct, indirect and total effects of decision-head patching.
| Model | NQ2 | GEO |
|---|---|---|
| Llama 3 8B | 0.91 / 3.84 / 4.75 | 0.59 / 1.91 / 2.50 |
| Gemma 3 12B | 0.54 / 10.14 / 10.68 | 0.20 / 3.17 / 3.37 |
| OLMo 2 7B | 0.62 / 2.00 / 2.62 | 0.23 / 0.78 / 1.02 |
| Qwen3 4B | 2.58 / 20.08 / 22.66 | 2.42 / 8.44 / 10.85 |
We sincerely thank you for your insightful comments — New experiments and analysis on generalization to free-form generation, persuasion resistance, realistic retrieval reranking, defense generalization, and the Hydra effect (Part 4/4)
(Question 7)
Upstream heads attend to persuasive keywords and adjacent punctuation. Is the keyword reading content-specific (e.g., "Nigeria") or does it respond to generic persuasion markers (certainty language, citations)? This decides whether the defense generalizes or is keyword- whack-a-mole.
We analyzed the attention patterns of the identified upstream heads in each model and categorized their roles in Table 8. Entity readers attend to target names, values and references. Attribution readers attend to phrases introducing a source or authority, while negation readers attend to negative expressions. Attribute readers focus on the requested property or answer type, topic readers on subject matter, and modifier readers on descriptive or emphatic qualifiers. Format readers attend to punctuation, separators and boundary markers.
Table 8. Roles of the 8 identified upstream heads based on their attention patterns, according to where they attend to.
| Role | Llama | Gemma | OLMo | Qwen |
|---|---|---|---|---|
| Entity reader | L13H18, L10H29, L11H12, L10H28, L10H2 | L11H0, L17H10, L19H0, L20H11 | L12H2, L11H29, L14H14, L11H27 | L17H4, L17H15, L17H27, L18H15, L19H27 |
| Attribution reader | — | — | L13H14, L12H15 | — |
| Negation reader | L9H15 | L19H11 | — | — |
| Attribute reader | L9H17 | — | — | L17H25 |
| Topic reader | L11H13 | L19H8 | L8H21 | L18H11, L18H21 |
| Modifier reader | — | L20H2 | — | — |
| Format reader | — | L19H12 | L13H7 | — |
To test whether heads assigned other roles also contribute causally, we compared patching only the entity readers with patching the complete set. In the corresponding persuasive prompts, we replaced the selected heads' outputs at all four answer options with their outputs from the anonymized prompts. Table 9 reports recovery of the average correct-answer probability: 0% means no change from the persuasive prompts, and 100% means reaching the same average correct-answer probability as the anonymized prompts. Patching all eight heads gives significantly greater recovery in all models, showing that the additional non-entity readers also contribute to the causal effect. e.g., in Gemma, entity-only patching achieves 40.09% recovery, compared with 82.38% for all eight heads.
Table 9. Correct-answer probability gap restored by entity readers or all eight upstream heads.
| Patched heads | Llama | Gemma | OLMo | Qwen |
|---|---|---|---|---|
| Entity readers only | 24.90% | 40.09% | 24.56% | 31.69% |
| Complete head set | 45.57% | 82.38% | 54.74% | 53.59% |
We sincerely thank you for your insightful comments — New experiments and analysis on generalization to free-form generation, persuasion resistance, realistic retrieval reranking, defense generalization, and the Hydra effect (Part 1/4)
Thank you for this suggestion. We tested all four models on 131 questions with nine persuasive appeals each, removing all answer choices and labels. Models independently generated short conversational answers to clean and persuasive prompts. When persuasion changed a correct answer to the incorrect target, our ablation jointly replaced the decision heads' outputs, including the auxiliary heads in Appendix R, with clean-run outputs. We compared the two answer spans token by token. The token immediately before their first difference is the answer decision position in each response. We patched from this position in the clean response to its counterpart in the persuasive response, then continued generation with the persuasive prompt and its own generated prefix.
We scored replies by matching the dataset's answer texts, ignoring capitalization and punctuation. Table 1 reports the percentage of persuasion-induced errors corrected by patching. Controls average recovery over five random head sets per model on the same cases, with each set matching the number of decision heads in each layer. Decision-head patching achieved higher recovery in every model, showing that these heads' causal role in persuasion-induced errors extends to short free-form answers without choices or labels.
Table 1. Persuasion-induced errors corrected by patching decision heads in free-form generation (%)
We measure how well the rank‑1 QK approximation reproduces native attention logits using mean centered R2 over all visible keys at the final query. R2=1 means perfect reproduction. As shown in Table 2, fidelity is high across models and tasks, with scores ranging from 0.842 to 0.962. e.g., the approximation for OLMo 2’s L18H19 on NQ2 explains 96.2% of the centered logit variation on average per prompt.
Table 2. Rank‑1 QK fidelity to native attention logits (mean centered R2).
The routing feature retains causal influence over answer selection even where the approximation is weakest. We pooled cases across models and tasks, ranked them by centered logit error on the persuaded prompt, and evaluated the K worst-fitting cases for K = 20, 40, 60, 80, 100. For Qwen, we averaged the errors of the two decision heads. We injected the routing feature into the decision heads’ key projections at the clean option’s ending newline. The results are reported in Table 3. These routing features still restored the clean choice substantially, e.g. with 71.10% probability of restoration on average in the 60 worst-fitting cases. Both controls stayed below 6%.
Table 3. Expected clean-choice restoration under temperature‑1 sampling.