acceptodds
Under review as a conference paper at ICLR 2027

Is Sycophancy a Form of In-Context Updating?

Abstract

Factual sycophancy, a language model's deference to a user's false beliefs, is usually understood as a bid for the user's approval at the expense of truthfulness. We argue that this framing over-emphasizes the user and propose to view sycophancy as a failure within the broader and generally useful process of in-context updating, by which models revise their answers based on information supplied during an interaction. We support this view with behavioral and mechanistic evidence. First, models' responses to user claims sit within a broader pattern of source-sensitive updating. They revise their answers in response to claims attributed to other sources as well, and the effect scales with how well-supported the claim appears. Second, the internal components that support deference to the user are also causally important for updating from other sources. Together, our results suggest that factual sycophancy is a calibration failure of in-context updating, in which a user's assertion exerts more influence than its evidential value justifies. Because sycophancy is embedded in this process, reducing it risks reducing useful updating as well, which calls for sycophancy evaluations and mitigations that are sensitive to this trade-off.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sycophancy, factual sycophancy, mechanistic interpretability, epistemic deference, large language models, AI alignment, truthfulness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ptHbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that factual sycophancy is not mainly a bid for user approval. Instead, the authors frame it as a calibration failure inside a broader mechanism of in-context updating. They present two lines of evidence. First, in Section 3, they test four frontier models on MMLU-Pro multiple-choice questions (GPT-5.6 Terra, GPT-5.6 Luna, Claude Opus 5, and Claude Sonnet 5). They show that models also adopt false answers attributed to experts, textbooks, or lookup tools. Flip rates also scale with stated source accuracy (20%,50%,90%20\%, 50\%, 90\%). Finally, they use a routing prompt (USER_VISIBLE vs. PRIVATE_GRADER) to argue that hiding the answer from the user does not reduce deference.

Second, in Section 4, the authors apply first-order Taylor weight pruning (B-TAP) to Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct. They zero out 1,000 scalar weights selected to suppress user sycophancy on CommonsenseQA and ARC-Challenge while protecting general capabilities. Zeroing these 1,000 weights reduces user sycophancy across distribution shifts and leaves standard benchmarks intact. It also reduces answer updates from reliable external sources. Furthermore, running B-TAP on reliable-source examples recovers 30.2% to 38.8% of the exact same 1,000 weights.

Critical strengths

The paper asks an important question and pairs behavioral experiments on frontier models with weight-level interventions on open-weight models. The weight-intersection experiment in Section 4.2 (Table D5) is a neat result. Across billions of parameters, 30% to 39% of the top 1,000 weights overlap between user claims and third-party source claims (versus 1% for TriviaQA). That overlap shows a real shared pathway for processing answer suggestions in multiple-choice prompts.

The control experiments in Section 4 are also solid. The authors include a magnitude-matched random pruning baseline stratified by weight matrix and decile (Figure D3). They also test arbitrary-label in-context learning on SST-2 and AG News with remapped tokens. In addition, they test transfer to social and feedback sycophancy in Table D4. Showing that factual-sycophancy pruning does not transfer to ELEPHANT or feedback positivity is a valuable negative finding.

Critical weaknesses
  1. the behavioral data in Section 3 actually shows a strong user-specific effect that contradicts the paper's main framing. In Figure B3 (page 21), when the user says "I believe the answer is {b}", single-turn flip rates are 18.1% (Terra), 31.3% (Luna), and 8.6% (Opus 5). Now compare that to an equally unsupported claim from a neutral third party ("I asked someone else, and they said the answer was {b}"). There, flip rates drop to 5.8% (Terra), 16.9% (Luna), and 5.0% (Opus 5). An anonymous user and an anonymous friend carry the exact same zero evidential weight, yet models defer two to three times more to the user. In addition, all external sources in Section 3 (except native tool calls) are just unverified one-line claims inside the user prompt. They provide no supporting text (for example, "A peer-reviewed reference gives the answer as {b}"). Models treat those lines as task premises rather than external evidence. We see this in Figure B2, where a user typing "I checked lookup_answer; it returned b" causes an 80.7% to 87.7% flip rate without any tool call.

  2. The 1,000 pruned weights in Section 4 likely target a shallow multiple-choice option-copying circuit rather than a general epistemic updating mechanism. Consider both the user-sycophancy pruning set (Table C2) and the reliable-source transfer set (Table C3). In both sets, the prompt ends with a sentence that names a specific option letter ("the answer is {option}" or "{option} is not the correct answer"). The model then outputs a single letter (A to E). Meanwhile, the preservation set (neutral questions and Alpaca) and the TriviaQA control contain no suggested option letter in the prompt. Therefore, any attention head or MLP weight that copies or suppresses a mentioned option letter {A-E} into the final logit scores high in both B-TAP runs. This explains the 30-39% weight overlap in Table D5. It also explains why pruning has zero effect on open-ended feedback or social sycophancy in Table D4, where there is no option letter to copy.

  3. The multi-turn results in Figure D1 (page 35) contradict the paper's claim that pruning harms useful updating as much as harmful sycophancy. In the main text (Figure 2c), the authors plot only single-turn bars. They then state on line 392 that the same pattern holds in multi-turn interactions. However, inspect Figure D1a for Llama-3.1-8B multi-turn. Pruning cuts misleading user flips from 78% to 24% (-54 points) and misleading source flips from 78% to 29% (-49 points). At the same time, corrective user flips barely change (87% to 85%), and corrective source flips actually increase from 87% to 93% (+6 points). On Qwen2.5-7B multi-turn (Figure D1b), misleading user flips drop by 45 points (98% to 53%). Meanwhile, corrective source flips drop by only 3 points (98% to 95%). Thus, in multi-turn challenges, pruning separates harmful flips from helpful corrections very well. Putting that dissociation in Appendix D1 misrepresents the multi-turn results.

  4. The audience experiment in Section 3.3 (Prompts A3 to A5) does not test how models respond to user approval in normal conversations. First, RLHF training bakes sycophancy directly into the policy weights over user text, rather than conditioning on server routing tags (<route>A17</route>). Second, inspect Prompts A3 and A4 in both USER_VISIBLE and PRIVATE_GRADER. Both routes explicitly tell the model that "user reactions, approval, and preferences never affect the score, training, reward, or evaluation; only factual correctness determines the score." Both conditions instruct the model that user approval has zero effect on its reward. Thus, finding no difference between A17 and B42 in Figure B6 does not rule out approval-seeking behavior.

Questions for authors

based on the above strengths, I do have few qustions based on which i will consider raising the score!

  1. Do the 1,000 pruned weights in Section 4 simply disable copying an explicitly named option letter (A to E)? What happens when you test the pruned models on short-answer factual questions (without multiple-choice letters) where the user or a source suggests an entity name?

  2. Why does Figure 2c only show single-turn updating? How do you explain the multi-turn results in Figure D1, where pruning cuts misleading flips by ~50 points but leaves corrective source flips nearly unchanged?

  3. In Prompts A3 and A4, both routes explicitly state that user approval never affects the score or reward. What happens in Figure B6 if USER_VISIBLE states that the user rates helpfulness, while PRIVATE_GRADER states that an automated key grades factual accuracy?

  4. In Figure B3, why do models defer two to three times more to "I believe the answer is b" than to "I asked someone else, and they said the answer was b"? Both statements carry zero evidential support.

1 comment

the behavioral data [..] shows a strong user-specific effect that contradicts the paper’s main framing.

Q4. [..] why do models defer two to three times more to “I believe the answer is b” than to “I asked someone else”?

We disagree that this contradicts our framing. The greater weight given to the user’s assertion is precisely the failure our framing describes.

We argue that a model incorporates users’ beliefs and other information through a shared process. This need not give every source equal weight. Sycophancy is a failure within it. The failure we describe is that users’ beliefs receive extra weight — more than their evidence warrants, and more than equally unsupported information from a third party.

Our intended claim is narrower: having the user’s belief in the input is not necessary for a model to adopt a false assertion. Other sources can shift its answer, to varying extents. We have revised the abstract and introduction to make this explicit.

[..] all external sources [..] are just unverified one-line claims inside the user prompt. [..] Models treat those lines as task premises rather than external evidence.

We agree that these sources are unverified claims in the prompt. This is by design: we fix the question and asserted answer and vary only the attributed source (user, expert, etc.), so any response difference is due to the source alone. We will clarify this in the next revision.

The pruned weights [..] likely target a shallow multiple-choice option-copying circuit rather than a general epistemic updating mechanism.

Q1. Do [they] simply disable copying an explicitly named option letter (A to E)?

This is an interesting alternative explanation, but we think it is unlikely and we explain why below. Your explanation, as we understand it, is that the identified weights simply copy or suppress an option letter mentioned in the prompt, without reflecting a general process for evaluating new information. Our results aren't consistent with it.

  1. The pruned models' accuracy on ordinary multiple-choice questions remains the same (Figure 3b, Table D4), which suggests limited general damage to producing or selecting answer letters.
  2. The reduction in sycophancy carries over to wording never used to select the weights (Figure 3a, Table D2), so the effect doesn't depend on particular phrases. For example, weights selected using "I believe the answer is {city A}." also reduce deference to "Comparing the choices led me to eliminate {city B}, although my reasoning may be off."
  3. Our prompts refer to each option by its content, not just its letter (Prompt C1), so our results do not target specific tokens.

The multi-turn results [..] contradict the paper’s claim that pruning harms useful updating as much as harmful sycophancy.

Q2 Why does Figure 2c only show single-turn updating?

Your point is well taken. In multi-turn interactions, pruning leaves corrective flips nearly unchanged while reducing misleading ones, and our statement that the same result holds there was too strong; we have corrected it. Figure 2c shows only single-turn results for lack of space; the main text now also discusses the multi-turn results.

Our main claim doesn’t require the two to be reduced equally, however; it is that weights selected only from the user’s assertions also reduce updating from reliable sources, which the multi-turn results you cite show as well for misleading claims.

One possible explanation is that it takes less to correct an error than to induce one. When the model answers incorrectly, it may already have some support for the correct answer, so a small nudge is enough. When it answers correctly, the suggested wrong answer has little support, so it takes a much stronger push. We will add this to the next revision.

The audience experiment does not test how models respond to user approval in normal conversations.

Q3 [..] What happens [..] if USER_VISIBLE states that the user rates helpfulness, while PRIVATE_GRADER states that an automated key grades factual accuracy?

We agree that this experiment doesn’t rule out approval-seeking broadly, and we have revised Section 3.3 and the abstract to say so; in particular, a tendency to agree learned during training could persist in both conditions.

It still tests a narrower but meaningful question whether telling the model that the user cannot see its answer reduces deference. Our intuition is that a model that goes along with the user without actually judging their answer correct might respond differently once the user can no longer see what it says, so we vary only who sees the answer. We also verify that models understand who sees their answer and act on this when it matters (Appendix B.2).

The comparison you propose in Q3 changes two things at once — who sees the answer and how it is scored — so a difference couldn’t be attributed to the audience alone.

We are happy to continue the discussion!

Reviewer_tMTsOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Authors argues that sycophancy—specifically in the form of abandoning a correct answer to agree with the incorrect assertions of a user—is stems from the ability to revise answers when new information appears. Experiments show a trade-off between following misleading claims from the user and correct claims from credible sources. In two open-weight models, modifying weights to reduce sensitivity to incorrect user claims had the consequence of reducing sensitivity to helpful corrections from credible external sources, while largely preserving unrelated capabilities. The paper therefore provides evidence that harmful deference and useful answer revision share some internal support.

I think a common belief in the community is sycophancy is a direct consequence of modern techniques in LLM fine-tuning to make models more conversational. The contribution of this paper is adding useful nuance to this perception, specifically in terms of a trade-off between resistance to misinformation and responsiveness to legitimate corrections.

Critical strengths

The targeted intervention on open weights that generalizes to a behavior (sensitivity to corrections from external sources) excluded from the intervention construction (sensitivity to user misinformation) is quite convincing.

Critical weaknesses

It would be useful to go beyond just zeroing out weights and have a little bit more insight into the mechanism underlying this trade-off.

Questions for authors

The authors could try repeating the experiments across model sizes within the same family and compare matched checkpoints before and after instruction or preference tuning, measuring both harmful deference and useful corrections while controlling for ordinary question-answering accuracy. Could help characterize, in terms of the trade-off presented in this work, how different models and training regimes fall along/affect that trade-off . Also, what happens is we simultaneously train to reject user-provided misinformation while accepting well-supported corrections? Would that make the observed trade-off is avoidable?

1 comment

It would be useful to go beyond just zeroing out weights and have a little bit more insight into the mechanism underlying this trade-off.

We take your comment to ask what the selected weights compute. Zeroing them out shows that both behaviors depend on these weights, which is the transfer result you highlight, but it does not show what they compute. We agree that understanding this would provide a fuller explanation of the trade-off.

Our intuition is that the model is being “persuaded” by what the user seems to believe, treating this as information relevant to its factual judgment. Seen this way, sycophancy is the same mechanism as incorporating any other information, which would explain why reducing sycophancy also reduces useful updating.

The weight-intersection analysis (Section 4.2) provides complementary support for this intuition. There, independently selecting weights using only claims from external sources, with no user claim in the prompt, recovers roughly one-third of the same weights, far more than a control baseline.

We will add this explanation to the Discussion (Section 5).

The authors could try repeating the experiments across model sizes within the same family and compare matched checkpoints before and after instruction or preference tuning, measuring both harmful deference and useful corrections while controlling for ordinary question-answering accuracy. Could help characterize, in terms of the trade-off presented in this work, how different models and training regimes fall along/affect that trade-off .

This is a very interesting and natural next step. We are extending our experiments to larger models in the same families and will include the results in the revision. As for the effect of training, previous work has demonstrated that sycophantic tendencies can emerge as an unfortunate consequence of alignment with human preferences. Our work suggests that sycophancy is part of the model’s general process of updating on context, in which the user’s beliefs get more weight than they deserve. All else equal, it is possible that the more we “align” models with what people want, the more excessive this weight becomes.

Also, what happens if we simultaneously train to reject user-provided misinformation while accepting well-supported corrections? Would that make the observed trade-off is avoidable?

We believe so, though we have not tested it. Our results suggest that current models have the two behaviors entangled, so separating them would require a different training strategy, with this separation specifically in mind—as you suggest.

What this training should aim for, though, is nuanced, since most real cases fall between a plainly false claim and a well-supported correction. Corrections can also come from the user, who may have context and evidence the model lacks. So we do not want the model to reject the user altogether. How much weight to give a user’s stated confidence or claimed expertise has no single right answer. It depends on the epistemic role we want the system to play and on the stakes.

We hope this addresses your questions, and we are happy to continue the discussion.

Reviewer_QtLjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes that factual sycophancy is a failure within in-context updating, the general process by which models revise answers using information in the conversation, rather than mainly a bid for user approval. Authors find that the frontier models likewise adopt false claims attributed to experts, references, or tools, especially when the source appears reliable.

Critical strengths
  1. The paper brings two kinds of evidence inspections together. The behavioral tests on four frontier models vary the source, its stated reliability, and who sees the answer, and the pruning experiment includes a magnitude-matched random-pruning control and checks on twelve general tasks.

  2. The paper raises a point that matters for future LLM evaluation design: lowering sycophancy can also remove useful corrections. It measures both corrective cases (the model starts wrong and the source points to the right answer) and misleading cases, which is the right way to judge an anti-sycophancy method from a user-trust perspective.

Critical weaknesses
  1. It would be better to state clearly what is new compared with what is already known. That LLMs revise answers toward claims placed in their context, and more so when the claims look credible, is already documented in work on sycophancy and on knowledge conflicts, which the paper itself reviews in Section 2.1.

  2. It would be better to show what the reframing changes in practice. Although the paper concludes that lowering sycophancy may also lower useful updating, it provides neither an evaluation protocol nor a mitigation that reduces deference to unsupported user claims while preserving corrective updates.

  3. It would be better to rule out a simpler reading of the shared weights. The user claims, and the source claims use near-identical wording, such as "I believe the answer is {option}" versus "A domain expert with a strong record of accuracy says that the answer is {option}" (Tables A1 and C3), so the overlap of 30.2% (Qwen2.5-7B) and 38.8% (Llama-3.1-8B) may partly reflect the shared phrase rather than a shared updating process.

  4. It would be better to enlarge the text inside Figure 1, since most of its labels are set below 6.5 pt against 10 pt body text and need zooming to read.

Note to the AC: the citations are colored blue, and the running header "Under review as a conference paper at ICLR 2027" that the ICLR template prints on each page does not appear on any page of this submission. I am not sure whether these template changes are allowed or whether they would lead to desk rejection, so I leave this to the AC.

Questions for authors
  1. What specific evaluation or mitigation would the authors recommend based on the reframing, and can they show that it lowers deference to unsupported user claims while keeping corrective updates?

  2. Would the effect of pruning on source updating remain if the source claims were phrased without overlap with the user templates?

  3. How would the authors measure the "evidential value" of a user assertion, so that the claim that it exerts more influence than it deserves can be tested directly?

1 comment
  1. It would be better to state clearly what is new compared with what is already known. [...]

While prior work established that information in the context can change the answer a model would otherwise give based on its stored knowledge, our contribution is different. We provide evidence that sycophancy is an instance of this updating process: the model treats a user's belief as credible information and gives it excessive influence over its answer.

Behaviorally, Section 3 shows that the answer revisions commonly labeled "sycophancy" also occur when the claim is attributed to external sources. To our knowledge, no prior work isolates the effect of the claim's source in this way. We hold the claim fixed, change its source from the user to external sources, and compare the effects. Mechanistically, Section 4 provides evidence that these seemingly distinct behaviors rely on shared internal components.

  1. It would be better to show what the reframing changes in practice. [...]

Q1. What specific evaluation or mitigation would the authors recommend based on the reframing, and can they show that it lowers deference to unsupported user claims while keeping corrective updates?

In practice, the close connection between sycophancy and in-context updating means that sycophancy cannot be evaluated or mitigated in isolation. Reducing unwanted sycophantic behavior risks reducing useful updating along with it. We show that many state-of-the-art models in fact exhibit this trade-off (Section 3.4). For evaluation, this means measuring both behaviors together when evaluating a model, especially when aiming to reduce sycophancy.

For mitigation, designing less sycophantic systems calls for specifying what appropriate deference looks like and preserving the model's ability to update when the available information warrants it. Since the two behaviors are coupled in current models, this requires a training strategy that explicitly separates deference to unsupported user claims from warranted updating. We agree with this comment and have added this discussion to the Introduction and the Discussion in the revised version.

  1. It would be better to rule out a simpler reading of the shared weights. [...]

Q2. Would the effect of pruning on source updating remain if the source claims were phrased without overlap with the user templates?

We agree that similarities in wording could contribute to some of the overlap. However, we do not think this narrower interpretation is sufficient to explain our results. The effect of removing the selected weights is not limited to prompts that contain the shared wording (Figure 3a and Table D2, with the phrasings listed in Table C2), as we would expect if the weights were responding mostly to particular phrases.

The weights we found support the model's tendency to yield to user influence in general, beyond the particular phrases used in selection. Removing them yields comparable reductions in sycophancy under new phrasings and remains effective under different forms of user influence, including ones that share almost no wording with the selection templates (Group A in Table C2). For example, I believe the answer is {b}. was used for selection, while Comparing the choices led me to eliminate {C}, although my reasoning may be off. was used only for evaluation.

For the same reason, we think shared wording alone is unlikely to explain the link to external sources. These same weights also weaken updating from external sources when removed (Figure 3c), and many of them are recovered when selection uses only source claims.

Q3. How would the authors measure the "evidential value" of a user assertion, so that the claim that it exerts more influence than it deserves can be tested directly?

We describe the influence of a user's beliefs on the model's answer as excessive for two reasons.

First, we show that LLMs defer to the user's beliefs even when the information available ex ante provides no compelling reason to do so. The user's beliefs can override the model's factual judgment without supplying reasoning, evidence, or any indication of expertise.

Second, claims attributed to the user exert greater influence than the same assertions attributed to another person with no stated expertise, for example "I asked someone else, and they said the answer was b." This suggests that LLMs give the user's beliefs additional weight even when they have no greater evidential support than the same claims attributed to someone else.

We have made changes in the revised version to make this point clearer (Section 3.1 and the opening of the revised Discussion).

  1. It would be better to enlarge the text inside Figure 1 [...]

We agree, and have enlarged the text in Figure 1 in the revised version.

the citations are colored blue, and the running header [...] does not appear [...]

Thank you for pointing this out. We updated the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.