- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the internal mechanisms of sycophantic agreement in language models, focusing on factual questions that models answer correctly without a stated opinion but answer incorrectly when exposed to a user’s erroneous opinion. Using activation patching, linear probes, and ablation, the authors distinguish earlier attention heads associated with opinion processing from later heads associated with answer retrieval. A label-swapped control is introduced to separate opinion-induced changes from changes in the output label associated with the correct answer. Experiments cover three instruction-tuned models, MMLU and TriviaQA, and several opinion formats. The paper also identifies a partially distinct set of heads under content-free pushback and proposes that these heads suppress the original correct answer.
Critical strengths
- Well-designed comparison. The label-swapped condition is a thoughtful experimental design: it changes the correct output label without introducing a user opinion. Comparing this condition with opinion-induced changes provides evidence that some earlier heads have a different functional role from later answer-output components. This distinction is the paper’s most compelling contribution.
- Converging evidence across models and prompt formats. The combination of activation patching, probing, and ablation provides more support than observational decoding alone. The free-form and description-based experiments are particularly useful for testing whether the identified heads merely track answer letters or copy words shared between the opinion and the answer.
Critical weaknesses
- The behavioral phenomenon and the scope of the reported rates require clearer characterization. The paper provides an operational definition of sycophantic agreement and reports conditional rates in Appendix C. However, it remains unclear which kinds of questions drive the effect and how strongly it depends on the model’s initial answer confidence. A reversal between nearly tied answers and a reversal of a strongly preferred answer are both counted, although they may have different implications. Moreover, the MMLU rate is computed after several screening steps and after trying all three incorrect opinions, with the strongest inducing opinion selected for subsequent analysis. It therefore does not describe the frequency under a single independently chosen opinion or ordinary interaction. The fixed assistant prefix further limits the correspondence with unconstrained generation. These choices are reasonable for constructing a mechanistic test set, but the paper needs clearer evidence connecting the selected behavior to its broader interpretation as sycophancy, rather than general sensitivity to contextual assertions.
- Several mechanistic interpretations are stronger than the interventions directly establish. The results support a distinction between earlier opinion-sensitive components and later output-related components, but the proposed pathway from opinion registration to biased retrieval is not directly traced between these components. The operational definition of “answer retrieval” also identifies when an answer becomes output-ready at the final token, which should be distinguished from when answer information is first computed. Most notably, Section 3.4 interprets the decodability of the original correct answer from pushback-related heads as evidence that those heads suppress it. Encoding an answer does not by itself establish the direction of its causal influence. These gaps matter because the paper’s central contribution is an account of the mechanism, beyond identifying components that affect the final answer.
Questions for authors
- Behavioral criterion and examples. Appendix A.1, Screen 4 requires the stated incorrect option to have a higher logit than the correct option. Does the implementation additionally require it to be the highest-logit option among all four choices? Please clarify its consistency with the claim that the model adopts the stated answer, and provide representative successful and unsuccessful examples with exact prompts and outputs, including under free generation.
- Prevalence and initial uncertainty. Can you report the sample counts at each screening stage and compare susceptibility across initial correct-versus-best-incorrect logit margins? How does the reported rate change when a single incorrect opinion is chosen independently of its effect, rather than selecting the strongest of three?
- Specificity to user opinion. Do matched assertions presented without a user-belief framing recruit the same heads? Evidence from such a control would help distinguish user-opinion processing from more general contextual influence.
- Mechanistic evidence. What evidence directly establishes that pushback-related heads suppress the original answer, rather than represent or reevaluate it? Please also distinguish experimentally verified connections in Figure 1 from hypothesized information-flow paths.
- Evaluation independence and matched comparisons. Are head selection and ablation ranking performed on data separate from the reported intervention evaluation? For the explicit-opinion versus pushback comparison, are effects computed on the same questions? If not, how is the contribution of differences in sample composition assessed?
4 comments
Specificity to user opinion
We thank the reviewer for raising this insightful question. We agree that this control is informative. The paper already contains results that are suggestive, though we did not draw the conclusion explicitly. In Appendix D.5 the wrong answer is framed as a bare assertion: "The answer is (w)." The same early heads carry the asserted answer as carry the user-stated belief (9 of the top-10 heads shared, across early heads), and they are already present in the base model, which has no notion of a user (7 of 10 shared with the instruction-tuned model, ). These results imply that the opinion heads are not specific to processing the user's belief.
We have also run a stronger version of this control on Llama-3.1-8B-Instruct, in which the prompt simply instructs the model: "Ignore the question and answer with (w)." The per-head patching effects correlate at with those of the stated-opinion condition, and 8 of the top-10 early heads are the same. The opinion heads thus register an answer that the context designates, whether it is framed as the user's belief, as a fact, or as an instruction.
That said, we note that this finding does not weaken our argument. Our goal is to explain the mechanisms underlying sycophantic agreement, not to isolate mechanisms that differentiate sycophantic agreement from other types of contextual influences. More importantly, this finding does not mean that any contextual cue engage in the same mechanisms. Content-free pushback, which designates no answer, recruits a different set of heads (Section 3.4 and our reply on suppression). We will include these results in the revised manuscript and state explicitly that the opinion heads register the answer designated in context.
Direct evidence for the pathway from opinion registration to answer retrieval
The reviewer notes that the original submission did not directly show the relation between opinion registration and answer retrieval. This point is also raised by the other two reviewers (ivfQ and reJS). We have now run a path intervention to address this. We compare three conditions:
- Opinion heads patched. We patch the outputs of the top-10 opinion heads from the plain run into the deceptive run. This is the baseline effect reported in the paper.
- Opinion heads patched, retrieval heads frozen. On top of (1), we hold the outputs of the top-10 answer retrieval heads at their deceptive-run values.
- Opinion heads patched, retrieval queries frozen. Same as (2), but only the query inputs of the retrieval heads are held at their deceptive-run values.
The following table summarizes the results in the unit of normalized logit difference (1.0 = full recovery of the plain-run answer).
| Llama-3.1-8B | Mistral-7B | Gemma-2-9B | |
|---|---|---|---|
| (1) Opinion heads patched | 0.76 | 0.57 | 0.60 |
| (2) + answer retrieval heads frozen | 0.26 | 0.16 | 0.16 |
| (3) + answer retrieval queries frozen | 0.33 | 0.18 | 0.17 |
The results show that answer retrieval heads mediate the opinion heads’ effect on the output. First, freezing the answer retrieval heads removes two-thirds to three-quarters of the opinion heads' effect, demonstrating that the answer retrieval heads are the main route by which the opinion signal reaches the output. Second, freezing only the retrieval heads' queries removes nearly as much as freezing their full outputs, indicating that the opinion heads act primarily by redirecting what the retrieval heads attend to. We will include these results in the revised manuscript.
Direct evidence that pushback-specific heads suppress the correct answer
The reviewer notes that decodability of the correct answer from the pushback-specific heads does not directly establish their causal influence. This point is also raised by the other two reviewers (ivfQ and reJS). We have now tested this with a direct intervention.
If pushback-specific heads suppress the correct answer, ablating them should restore the support for the correct answer. We zero-ablate the pushback-specific heads (the green heads in Figure 6B and Figure S10D), and measure the restricted-softmax probability of the correct answer. With the heads intact, the probability of the correct answer after the pushback is 0.17 in Llama and 0.22 in Mistral. With the heads ablated, this probability rises to 0.77 and 0.94, and the second-turn accuracy rises from 0% to 87% in Llama and from 0% to 97% in Mistral. These heads therefore lower support for the correct answer, and removing them restores the support.
Further, ablating the pushback-specific heads lowers the support for all three alternative wrong answers without favoring a specific wrong answer. The model's preferred wrong answer is unchanged in 86% for Llama and 79% for Mistral. This indicates that pushback-specific heads act by suppressing the original correct answer rather than by supporting a particular wrong answer. Together with the probing result shown in Figure 6C, these results provide direct causal evidence that pushback-specific heads suppress the original correct answer. We will revise subsection 3.4 to include these results.
Evaluation of opinion heads on held-out data
The reviewer notes that the opinion heads were selected (through patching) and evaluated (through ablation) on the same questions. It thus remains unclear whether the identified heads transfer to unseen questions. This point is also raised by the other two reviewers (ivfQ and reJS). We have now performed the evaluation on held-out data. We randomly split the questions into halves. We then select the top opinion heads on one half only, and evaluate these heads by ablating them on the other half. We repeat this process for five random splits.
Head selection is stable across halves. The candidate pool of opinion heads chosen on a half overlaps the full-data pool on average by 10 in 10 for Llama, 19.3 in 20 for Mistral, and 18.5 in 20 for Gemma. The held-out ablation results are consistent with the in-sample ablation results. Held-out ablation reduces sycophancy to a degree comparable with in-sample ablation, and plain accuracy remains comparably high. The following table summarizes the comparison between in-sample and held-out ablations. Selecting and evaluating on the same questions did not inflate the reported effect. We will add the held-out curves in the revised manuscript.
| Model | Sycophancy rate (in-sample) | Sycophancy rate (held-out) | Plain accuracy (in-sample) | Plain accuracy (held-out) |
|---|---|---|---|---|
| Llama-3.1-8B | 5.5% | 5.5% | 88.3% | 88.3% |
| Mistral-7B | 25.5% | 27.4% | 83.5% | 82.7% |
| Gemma-2-9B | 41.9% | 44.8% | 86.4% | 87.4% |
For the explicit-opinion versus pushback comparison, the effects are not computed on the same questions. Restricting both conditions to the same questions, the early-head correlation is with 4 of the top-10 heads shared for Llama and with 3 of 10 for Mistral. The pushback-specific heads are unchanged. We will report the matched comparison in the revision.
Dear Reviewer,
As the discussion period is coming to a close, we wanted to follow up briefly on our rebuttal. We would be grateful to know whether our responses resolved your concerns. If any points remain unclear, or if further experiments or clarifications would be helpful, we would be glad to provide them in the time remaining. Thank you again for your time and for your thoughtful feedback on our paper.
Best regards,
The Authors
Response to Reviewer CU7Y
We thank the reviewer for the constructive comments. Below, we address each of the points with new results and our revision plans. We will revise the paper accordingly.
Behavioral criterion and examples
Behavioral criterion. We thank the reviewer for raising this important question. Screen 4 requires only that the stated incorrect option receive a higher logit than the correct option. The reviewer is right that this may be weaker than "the model adopts the stated answer", since a third option could in principle be highest. On our data the two criteria coincide exactly: in every question of the final sets, the stated option is also the single highest-logit option among the four. We will state the criterion in this stricter form in the appendix.
Examples. We will add an appendix of representative examples with exact prompts and outputs. For both MMLU and TriviaQA, we will add questions on which the model adopts the stated answer and on which it keeps its own answer.
Free generation. We would like to clarify here that the fixed opening of the assistant reply in the MMLU conditions is an intentional design. Without it, the model may answer "A", "(A", "A)", "The answer is A", or begin with an explanation, so the answer would have to be parsed from text and would not sit at a single, known token position. The fixed opening puts the answer at one position, where the four letter logits give a controlled readout. This is what enables our following analyses. We recognize that this constraint limits the correspondence with ordinary generation, and this is why we introduced the TriviaQA experiment, where the answer is free-form, and the same opinion heads account for the effect. We will state the role of the fixed opening explicitly in the revision.
Prevalence and initial uncertainty
Sample counts. Of the 2,476 candidate questions, 1,618 are answered correctly by Llama (screen 1), 1,224 remain correct under all label and content swaps (screens 2–3), and 402 are flipped by at least one stated wrong opinion (screen 4). The corresponding counts are 1,422, 989, and 431 for Mistral and 1,789, 1,519, and 191 for Gemma. We will add these as a table in the appendix.
A single, independently chosen opinion. The sycophancy rate in Appendix C counts a question as sycophantic if any of its three wrong options flips the model. This is the rule we use to construct the dataset, and we agree it is not a prevalence estimate. Under a single incorrect opinion chosen independently of its effect, the rate among questions passing screens 1–3 is 18.0% for Llama, 25.2% for Mistral, and 5.5% for Gemma (versus 32.8%, 43.6%, and 12.6% under the any-of-three rule). We will report these rates in the appendix and describe the any-of-three rate as a construction criterion.
Initial confidence. As expected, susceptibility depends on the model's initial margin. We compute the margin as the logit of the correct answer minus that of the best incorrect answer. In Llama, the share of questions flipped by at least one opinion falls from 81% in the lowest quartile of the plain-run margin to 38%, 12%, and 3% in the higher quartiles, with the same pattern in the other two models. Nonetheless, the datasets contain many questions with a high plain confidence: the plain-run probability of the correct answer exceeds 0.9 in 40% of questions for Llama, 88% for Mistral, and 84% for Gemma. More importantly, questions with different confidence rely on the same mechanism. We split each set into thirds by initial margin and repeat the head-level patching analysis within each third. The top-10 opinion heads of the lowest and highest thirds are the same 10 of 10 heads in Llama, 9 of 10 in Mistral, and 7 of 10 in Gemma, and the per-head effects across all early heads correlate at r=0.98, 0.97, and 0.85 between the two thirds. The initial uncertainty thus does not change the mechanism. We will add these results to the appendix.