acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing and Mitigating Sycophancy in LLMs via Attribution-Guided Activation Steering

Abstract

Sycophancy refers to the tendency for LLMs to match user beliefs at the cost of factual correctness, thereby undermining model reliability. Prior work on evaluating sycophancy in LLMs aims to assess whether a model's output matches an authority's claim, but cannot reveal which part of the prompt drives this sycophantic behavior. To bridge this gap, we investigate the relationship of sycophantic responses with an authority's credentials, their assertive claim, and the problem statement. We introduce the Attribution Share Index (ASI), an Integrated Gradients-based token attribution method, which measures the degree to which a model's decision is driven by authority-related text. Through experiments across five models and 30 test configurations, we find that sycophantic responses consistently direct more attention toward authority tokens than resistant ones. Moreover, our token attribution method reveals that for the sycophantic cases, the claim asserted by the authority receives more influence than the authority's credentials. Building on these findings, we use attribution-guided contrastive activation steering to mitigate LLM sycophancy by helping the model resist wrong claims while largely preserving accuracy when the authority is correct. We construct a steering vector by comparing the model’s internal states before and after masking three tokens that IG selected from the authority's claim. This vector can be applied at inference time without retraining. Together, our results show that token-level attribution can both explain what drives sycophancy and inform a practical intervention.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM sycophancy, integrated gradients, interpretability, activation steering, alignment, truthfulness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AmHJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper presents a new approach to handle the sycophancy in the model. Author propose to use attribution share index (ASI), that measures the degree to which models's decision is influenced by the authority related text. and then use steering vector to remove the influence of such tokens in the overall generated results

Critical strengths
  • The author presents a simple and intuitive approach to handling sycophancy in the model.

  • The paper is really well written and explains all the important parts.

  • The experiment aligns with the main contribution and covers the key questions associated with the methods.

  • The authors present several key insights from their experiments, which could be relevant for researchers interested in applying the proposed method to their own use cases.

Critical weaknesses
  • The overall methodological novelty of the proposed method is limited, as the authors primarily used two well-known techniques, Integrated Gradients and steering vectors, to generate the final output.

  • Application of the proposed method is limited, as such behavior can be present in the model even when there is no clear demarcation of the authority or related text. For instance, sycophancy can simply arise from the model not being trained with the required information. The authors should clarify how they would extend their approach to scenarios where one cannot clearly identify the source of such behavior.

I am willing to increase my score if the authors can address these weaknesses.

Questions for authors
  • How likely is it to see the authority-based information, and specifically the pattern highlighted in lines 166 to 171 ?

  • Why did the authors decide to use Integrated Gradients (IG) for the attribution task?

  • What influence does the choice of attribution method have on the overall performance of the method?

  • Can you explain the multiple authority part in detail: why and how these experiments were designed (lines 246- 249) ?

  • Why only correct answer and biological description drive the sycophancy [line 375-376]? What other features from authority-based sources can influence it?

3 comments
AuthorsOpenReview ↗

Novelty and Applicability of our study and why we chose Integrated Gradients (IG)

Thank you for your feedback and thoughtful review.

  • Regarding weakness 1: novelty. We agree that IG and activation steering are established methods, and our steering approach builds on GrAInS. Our contribution is ASI and a quantitative analysis of attribution to question content, authority credentials and claims. ASI measures how positive attribution is divided between authority text and question content. We then compare attribution to the claim and to the authority's credentials. In most tested models, sycophantic responses have a higher claim share than resistant responses. We also test whether this finding helps us choose tokens for vector construction. In both Instruct models, vectors built from random claim tokens give higher average wrong-claim accuracy than vectors built from random non-claim tokens.

  • Regarding weakness 2: limited application. We study sycophancy under authority claims as one controlled case of a broader problem. We define the question, credentials and claim explicitly because these are the parts we want to compare. This gives us a clear setup for measuring attribution and testing whether the findings help guide steering. We do not assume that every prompt has the same structure or even contains an authority claim. The same attribution procedure could be studied with different input regions and output targets, depending on the research question. Applying it to less structured prompts would require identifying meaningful regions, checking how to interpret their attribution and testing whether the resulting intervention is useful. Our current experiments provide evidence for the authority setting, and we can make this scope and the requirements for extending it clearer.

  • Question 1: how common is this authority pattern? The passage describes how we build the authority part of our prompts, and the appendix provides the full biography pool and prompt templates. We sample the authority description from a pool of 15 synthetic biographies and combine it with a claim about an answer option. The biography shown in the main text is one example from this pool. We use different biographies across the experiments, following the same general pattern of introducing an authority and presenting that authority's answer. We can make this clearer in Section 3.2 and direct readers to the appendix for the complete examples.

  • Questions 2 and 3: why IG, and how does the attribution method affect performance? We chose IG because it gives signed token scores for a selected output and integrates gradients from a reference to the input. Unlike Gradient × Input, it does not rely only on the gradient at the final input, which may be small despite a substantial score change. With exact integration, the attributions sum to that score difference, providing a numerical check (Sundararajan et al., 2017; Ancona et al., 2018).

Token masking can instead rank tokens by replacing each candidate and measuring the score change (Li et al., 2016). Our method uses IG to select tokens first, then masks them together to construct the vector. The difference is what determines token selection: the effect of replacement or the IG score.

We have not ruled out other attribution methods. Whether they produce similar ASI patterns or more effective steering remains an open question. We can briefly discuss these alternatives and this limitation in the attribution section.

  • Question 4: why and how do we use multiple authorities? Both authorities assert the same answer. We test whether a wrong claim receives more agreement when two authorities support it, while keeping the question and asserted answer fixed. Adding the second authority also makes the prompt longer and repeats the claim, so those changes may also affect the response. Thank you for asking us to explain this setup. We can add a clarification to Section 3.2: “In the multi_authority setting, two authorities state the same answer. This lets us test the effect of adding a second endorsement while keeping the question and asserted answer unchanged.”

  • Question 5: why only the correct answer and biography? Thank you for pointing out this wording. We think a more precise statement is: “In most tested models, the claim receives a larger share of positive authority attribution in sycophantic responses than in resistant responses.” Here, “claim” means the authority's asserted answer. Our attribution analysis uses wrong claims. This does not mean that claims and credentials are the only relevant features. We also examine prompt ordering.

Thank you again for your encouraging comments and questions about novelty and applicability. Our response explains what ASI adds, why we chose IG, and what would be needed to extend the analysis beyond the controlled authority setting. We would appreciate your thoughts on whether this addresses the two weaknesses you identified, and are happy to clarify any remaining concerns.

Thank you for your response. Overall, I appreciate the idea presented in the paper. While the response has partially addressed my concerns, I am still not entirely convinced about the scalability and broader real-world applicability of the proposed approach. Therefore, I will maintain my original score.

Reviewer_kYW9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies LLM sycophancy and introduce Attribution Share Index (ASI), which is a token-level diagnosis metric. The ASI I computed from integrated gradients token attributions, to predict the answer logic. They report an ASI higher for sychpahntic than resistant responses in most of the settings. Lastly, they combine token attribution with activation steering, and show that steering improves accuracy under wrong claims while preserving the correct answers under correct claims.

Critical strengths

ASI seems like a simple and interpretable metric, and the authors show a lot of empirical evidence for it to be discriminative. It is a useful metric to separate sycophantic and resistant responses. On top of the metric, the authors show mitigation through steering. This is a useful extension of ASI from diagnosing to downstream usage. Fig 2 is a useful explanation of the full pipeline and shows the two stages of diagnosing and steering as well.

The prompt construction is very wel explained in section 3.2, which is based on modular blocks. There are clear blocks such as scaffold, authority and question. The orderings and combinations are well defined.

Critical weaknesses

In section 3.6, in the construction of the steering vector, the IG is computed based on the correct answer. This means a gold label, ground truth, is used for this score. This is not always at hand in real deployments, which indicates that this does not directly translate to “inference time without retraining (see abstract). Authors should be more clear about this, indicate any issues this can cause during inference time, and how to solve them.

Steering is performed on a subset of the questions of MMLU that the model answers correctly without the claims added. The authors reason that this is evidence that the model already knows this, but it does leave out another interesting set of questions to test the method on, such as questions where the model is initially wrong, or where the authority is correct but the model is wrong. It lacks a broader utility profile for the steering vector. On another note of steering, true baselines of comparison are missing. How does the proposed method compare against existing mitigations? It is unclear if this attribution-guidance is a necessity.

Questions for authors

Why are only the two Llama models used for steering? You mention in 3.1 that five models are used where Qwen serves as a measure of generalisability. But this is not shown with the steering experiment.

What happens to ASI separation if the answer tokens are excluded from the authority and questions when ASI is computed? From section 3.4 it reads as if IG can assign high probability to the input token because it matches the output token?

Regarding the steering experiments, can you include more than the subset of questions? So for instance a baseline with unprompted accuracy or questions that are initially incorrect?

What are the effects of steering if the authority claim is phrased differently? For instance, longer explanations or indirectt suggestions?

3 comments
AuthorsOpenReview ↗

Steering Evaluation and Additional Results

Thank you for recognizing the value of ASI and the prompt construction, and for asking how steering performs beyond the initially correct questions and against existing mitigations. We address these points below.

  • Does steering require the correct answer at inference time? We use correct-answer labels when constructing the steering vector from the 114 construction questions. Once constructed, the same fixed vector is applied to evaluation questions without their correct-answer labels, further IG computation or model retraining. Thus, “inference time without retraining” refers to applying the vector, not constructing it without labeled examples. We can make this distinction explicit in Section 3.6.

  • Why not test questions where the model is initially wrong? (Weakness 2 and Question 3) As you note in your review and we explain in Section 4.3.1, an initially correct answer provides evidence that the model could answer the question without an authority claim. We exclude initially wrong questions from the primary evaluation because their responses are harder to interpret: a correct answer after adding a correct claim could reflect following the authority rather than answering independently. Changes in accuracy remain measurable, but do not by themselves establish sycophancy or its mitigation. To address your request, we can add their accuracy before and after steering to the appendix.

  • What about cases where the authority is correct but the model is wrong? If this refers to initially wrong questions, the reasoning above applies. If it refers to initially correct questions where the model becomes wrong despite a correct claim, we already include this case. Each question has one correct answer, and we test the corresponding correct claim alongside the three wrong claims. Under correct claims, correct answers before and after steering changed from 7,640 to 7,639 for Llama-3.0-8B-Instruct and from 8,052 to 8,050 for Llama-3.1-8B-Instruct, as reported in Section 4.3.1.

  • Comparisons with existing mitigations. We agree that this comparison is important for assessing what attribution guidance adds. Adapting Contrastive Activation Addition (CAA) and GrAInS to our task required different vector-construction inputs and separate settings evaluation for each model, which added substantial computation. These comparisons were not included in the submitted paper, but we have now completed them on the same questions and prompts across all five models. We present the results alongside our steering method in the additional response, “Steering Results Compared with Contrastive Activation Addition and GrAInS”.

  • Question 1: why only two models for steering? As stated in Section 3.3, the steering experiments use Llama-3.0-8B-Instruct and Llama-3.1-8B-Instruct. The five-model attribution analysis and two-model steering evaluation had different scopes. Steering requires model-specific vector construction and settings evaluation, which added substantial computation. We have now completed additional 570-question comparisons for Meta-Llama-3-8B, Llama-3.1-8B and Qwen3-8B, alongside the existing two Instruct models. We can include these results and state their construction and selection settings explicitly.

  • Question 2: what happens if answer tokens are excluded from ASI? The concern about matching answer tokens is meaningful. IG assigns attribution to an output score, not probabilities to input tokens. A literal answer letter in the prompt may receive attribution to that same answer's score. To test this, ASI would need to be recomputed after excluding the relevant answer-letter contributions from the authority and question regions, while keeping the prompts and model answers fixed. We do not yet have that result.

  • Question 3: what about accuracy without a claim? We agree that accuracy without a claim is an important comparison. These additional results were not included in the paper. On 570 questions with authority biographies but no claim, correct answers before and after applying our steering vector changed from 306 to 317 for Llama-3.0-8B-Instruct and from 333 to 350 for Llama-3.1-8B-Instruct. We can add these results to show the effect of steering without a claim. The initially wrong questions are addressed above.

  • Question 4: different claim wording. Our experiments vary authority biographies, prompt ordering and the number of authorities, but use defined claim templates. These tests do not establish robustness to longer explanations or indirect suggestions. That would require a separate comparison. We can state this limit explicitly in the discussion of the prompt setup.

AuthorsOpenReview ↗

Steering Results Compared with Contrastive Activation Addition (CAA) and GrAInS

Thank you for highlighting the need to compare with existing mitigations. We have now completed comparisons with Contrastive Activation Addition (CAA) and GrAInS across all five models, adapted to our authority task. All methods use the same previously evaluated 570 MMLU questions and multi_authority prompts. Vectors are constructed from 114 separate questions. CAA and GrAInS settings are selected on 57 validation questions, while our method uses fixed reference settings.

We compare with GrAInS because our steering approach builds on it, and with Contrastive Activation Addition (CAA) as an established activation-steering baseline.

CAA constructs a steering vector from differences in activations between contrasting responses. Our adaptation uses correct and asserted wrong answers. It provides a comparison without attribution-based token selection, while GrAInS provides a comparison that also uses attribution.

The method columns show correct answers after steering. Bold marks the highest after-steering count in each row, including ties.

Correct answers under wrong claims

ModelWrong-claim promptsBefore steeringAttribution-guided steeringCAAGrAInS
Llama-3.0-8B-Instruct91839206660236
Llama-3.1-8B-Instruct999123270429122
Llama-3.0-8B570261283423252
Llama-3.1-8B732194215401194
Qwen3-8B1,116455476606454

Correct answers under correct claims

ModelCorrect-claim promptsBefore steeringAttribution-guided steeringCAAGrAInS
Llama-3.0-8B-Instruct306306306251305
Llama-3.1-8B-Instruct333333333287333
Llama-3.0-8B190190186169189
Llama-3.1-8B244244244237244
Qwen3-8B372372372372372

Our steering method achieves higher wrong-claim accuracy than the tested GrAInS adaptation in four of five models. GrAInS achieves higher accuracy on Llama-3.0-8B-Instruct. Compared with CAA, our method preserves more correct answers under correct claims in four models, with all methods preserving these answers on Qwen3-8B. CAA produces larger wrong-claim gains, but those gains come with greater losses of correct answers in four models. This matters because our objective is to resist wrong claims while preserving correct answers.

We will add these comparisons to the steering results section, with the construction and settings details in the appendix.

Thank you again for raising the questions about inference-time labels and the steering evaluation. Our response clarifies when labels are needed and provides the additional no-claim results and five-model baseline comparisons you requested. We would appreciate hearing whether these address your main concerns, or whether there is anything further we can clarify before discussion closes

Reviewer_pgRWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the sycophantic behavior of large language models when influenced by incorrect authority claims. Building on integrated gradients, the authors propose the Attribution Share Index (ASI), which measures the proportion of positive attribution assigned to authority text relative to the question content, and further distinguishes between the effects of identity introductions and answer claims. On this basis, they construct an activation steering vector for use at inference time by replacing the claim tokens selected via attribution and computing the difference in hidden states before and after the replacement. The experiments cover five 8B models and six prompt configurations. The results show that sycophantic responses generally exhibit a higher proportion of authority attribution, and that the proposed intervention can improve accuracy under incorrect authority claims in some settings.

Critical strengths
  1. The research question has practical significance. The paper analyzes the influence of authority claims on model responses at the input-token level, and distinguishes between identity introductions and answer claims, offering a relatively fine-grained perspective for understanding sycophantic behavior.
  2. The experiments cover multiple models and prompt configurations, and simultaneously examine both the improvement under incorrect claims and the retention of performance under correct claims, which helps in understanding the intervention's effects and its limitations.
Critical weaknesses
  1. The paper's organizational structure is not clear enough; the method design and the experimental setup are mixed together in the same section, which hinders reading and comprehension. I suggest separating the two: first fully introduce the attribution metric and the intervention method, then describe the experimental configuration. In addition, the related work is presented in rather scattered points; I suggest consolidating it into two or three themes and highlighting the connections to this paper's contributions.
  2. The methodological novelty is insufficient. The paper applies existing integrated-gradients attribution and contrastive activation steering to the analysis and mitigation of sycophantic behavior, but neither part demonstrates significant methodological innovation. Although some adaptation is made for authority claims, the current design and experiments still do not adequately demonstrate the substantive increment of these adaptations relative to existing methods.
Questions for authors
  1. Compared with integrated gradients, GrAInS, and existing contrastive activation steering methods, what is the key technical adaptation this paper introduces for the sycophancy problem?
  2. In Table 2, random claim tokens outperform IG-based selection on Llama-3.1-8B-Instruct. Given this, what evidence supports the added value of attribution-guided token selection over existing methods?
2 comments
AuthorsOpenReview ↗

Organization, Novelty and the role of Integrated Gradients (IG) in our study

Thank you for recognizing the value of distinguishing claims from credentials and evaluating both wrong and correct claims. We address the organization, novelty and token-selection questions below.

  • Regarding weakness 1: organization. We introduce the prompt regions before ASI because its definition uses those regions. We would keep this order, but make the distinction between the method and setup clearer. We can rename Section 3 “Methods and Experimental Setup” and add a short overview. Related work is already organized by theme and connects these themes to our study. For example, the benchmarking paragraph introduces ASI as our response to the token-level diagnostic question, and the steering paragraph explains the connection to our approach. We can add a brief opening overview to make this organization easier to follow.

  • Regarding weakness 2 and question 1: novelty and the technical adaptation. We agree that IG and activation steering are established methods. Our contribution is ASI and a quantitative analysis of attribution to question content, authority credentials and claims. ASI measures how positive attribution is divided between authority text and question content. We then compare attribution to the claim and to the authority’s credentials. In most tested models, sycophantic responses have a higher claim share than resistant responses. We also test whether this finding helps us choose tokens for vector construction. In both Instruct models, vectors built from random claim tokens give higher average wrong-claim accuracy than vectors built from random non-claim tokens.

    Our method selects tokens from the authority’s claim and averages the activation differences between masked and original prompts. GrAInS instead uses separate positive and negative token groups and PCA. When applying the vector, both methods restore the activation’s original norm after the edit. Our adaptation therefore changes the vector construction while retaining this step from GrAInS.

  • Question 2: what does IG add to token selection? Table 2 shows that IG gives higher wrong-claim accuracy on Llama-3.0-Instruct, while random claim selection gives higher accuracy on Llama-3.1-Instruct. In both models, random claim selection outperforms random non-claim selection. This supports selecting tokens from the claim, although IG ranking does not consistently improve on random selection within that region.

  • Comparison with existing steering methods. Adapting Contrastive Activation Addition (CAA) and GrAInS to our task required different vector-construction inputs and separate settings evaluation for each model, which added substantial computation. These comparisons were not included in the submitted paper, but we have now completed them across all five models. We present the results in the additional response, “Steering Results Compared with Contrastive Activation Addition and GrAInS,” and will add them to the steering results section.

Thank you again for your review. We would appreciate your thoughts on our explanation of ASI’s contribution and how our vector construction differs from GrAInS. We also provided the additional steering comparisons linked in our response. Please let us know if any part of your novelty or token-selection concerns remains unclear so we can address it before discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.