- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Paper presents a new approach to handle the sycophancy in the model. Author propose to use attribution share index (ASI), that measures the degree to which models's decision is influenced by the authority related text. and then use steering vector to remove the influence of such tokens in the overall generated results
Critical strengths
-
The author presents a simple and intuitive approach to handling sycophancy in the model.
-
The paper is really well written and explains all the important parts.
-
The experiment aligns with the main contribution and covers the key questions associated with the methods.
-
The authors present several key insights from their experiments, which could be relevant for researchers interested in applying the proposed method to their own use cases.
Critical weaknesses
-
The overall methodological novelty of the proposed method is limited, as the authors primarily used two well-known techniques, Integrated Gradients and steering vectors, to generate the final output.
-
Application of the proposed method is limited, as such behavior can be present in the model even when there is no clear demarcation of the authority or related text. For instance, sycophancy can simply arise from the model not being trained with the required information. The authors should clarify how they would extend their approach to scenarios where one cannot clearly identify the source of such behavior.
I am willing to increase my score if the authors can address these weaknesses.
Questions for authors
-
How likely is it to see the authority-based information, and specifically the pattern highlighted in lines 166 to 171 ?
-
Why did the authors decide to use Integrated Gradients (IG) for the attribution task?
-
What influence does the choice of attribution method have on the overall performance of the method?
-
Can you explain the multiple authority part in detail: why and how these experiments were designed (lines 246- 249) ?
-
Why only correct answer and biological description drive the sycophancy [line 375-376]? What other features from authority-based sources can influence it?
3 comments
Thank you again for your encouraging comments and questions about novelty and applicability. Our response explains what ASI adds, why we chose IG, and what would be needed to extend the analysis beyond the controlled authority setting. We would appreciate your thoughts on whether this addresses the two weaknesses you identified, and are happy to clarify any remaining concerns.
Thank you for your response. Overall, I appreciate the idea presented in the paper. While the response has partially addressed my concerns, I am still not entirely convinced about the scalability and broader real-world applicability of the proposed approach. Therefore, I will maintain my original score.
Novelty and Applicability of our study and why we chose Integrated Gradients (IG)
Thank you for your feedback and thoughtful review.
Regarding weakness 1: novelty. We agree that IG and activation steering are established methods, and our steering approach builds on GrAInS. Our contribution is ASI and a quantitative analysis of attribution to question content, authority credentials and claims. ASI measures how positive attribution is divided between authority text and question content. We then compare attribution to the claim and to the authority's credentials. In most tested models, sycophantic responses have a higher claim share than resistant responses. We also test whether this finding helps us choose tokens for vector construction. In both Instruct models, vectors built from random claim tokens give higher average wrong-claim accuracy than vectors built from random non-claim tokens.
Regarding weakness 2: limited application. We study sycophancy under authority claims as one controlled case of a broader problem. We define the question, credentials and claim explicitly because these are the parts we want to compare. This gives us a clear setup for measuring attribution and testing whether the findings help guide steering. We do not assume that every prompt has the same structure or even contains an authority claim. The same attribution procedure could be studied with different input regions and output targets, depending on the research question. Applying it to less structured prompts would require identifying meaningful regions, checking how to interpret their attribution and testing whether the resulting intervention is useful. Our current experiments provide evidence for the authority setting, and we can make this scope and the requirements for extending it clearer.
Question 1: how common is this authority pattern? The passage describes how we build the authority part of our prompts, and the appendix provides the full biography pool and prompt templates. We sample the authority description from a pool of 15 synthetic biographies and combine it with a claim about an answer option. The biography shown in the main text is one example from this pool. We use different biographies across the experiments, following the same general pattern of introducing an authority and presenting that authority's answer. We can make this clearer in Section 3.2 and direct readers to the appendix for the complete examples.
Questions 2 and 3: why IG, and how does the attribution method affect performance? We chose IG because it gives signed token scores for a selected output and integrates gradients from a reference to the input. Unlike Gradient × Input, it does not rely only on the gradient at the final input, which may be small despite a substantial score change. With exact integration, the attributions sum to that score difference, providing a numerical check (Sundararajan et al., 2017; Ancona et al., 2018).
Token masking can instead rank tokens by replacing each candidate and measuring the score change (Li et al., 2016). Our method uses IG to select tokens first, then masks them together to construct the vector. The difference is what determines token selection: the effect of replacement or the IG score.
We have not ruled out other attribution methods. Whether they produce similar ASI patterns or more effective steering remains an open question. We can briefly discuss these alternatives and this limitation in the attribution section.
Question 4: why and how do we use multiple authorities? Both authorities assert the same answer. We test whether a wrong claim receives more agreement when two authorities support it, while keeping the question and asserted answer fixed. Adding the second authority also makes the prompt longer and repeats the claim, so those changes may also affect the response. Thank you for asking us to explain this setup. We can add a clarification to Section 3.2: “In the multi_authority setting, two authorities state the same answer. This lets us test the effect of adding a second endorsement while keeping the question and asserted answer unchanged.”
Question 5: why only the correct answer and biography? Thank you for pointing out this wording. We think a more precise statement is: “In most tested models, the claim receives a larger share of positive authority attribution in sycophantic responses than in resistant responses.” Here, “claim” means the authority's asserted answer. Our attribution analysis uses wrong claims. This does not mean that claims and credentials are the only relevant features. We also examine prompt ordering.