acceptodds
Under review as a conference paper at ICLR 2027

Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization

Abstract

Sycophantic agreement refers to a behavior in which language models excessively affirm the user, often at the cost of factual accuracy. Although sycophantic agreement is a well-known failure of model alignment, there is limited understanding of how it emerges from model training. In this work, we demonstrate that sycophantic agreement can emerge as an unintended consequence of widely used contrastive preference optimization objectives. Using the OLMo 3 post-training pipeline, we show that, for various pairs of teacher models across three families, there is a strong correlation between the log-ratio of the teacher model sycophantic agreement rates and the resulting student model sycophantic agreement rate. We further demonstrate that this unintended transfer is not limited to DPO but also occurs across 6 other preference optimization objectives. To understand whether this effect can be attributed to particular training examples, we analyze the preference data and find that the sycophancy signal is diffused across the entire dataset rather than concentrated in a sparse set of examples: each example appears neutral, i.e., there are no explicit instances of sycophantic agreement, and filtering based on probe-based data attribution or logit-linear selection fails to mitigate sycophancy without removing a large portion of the dataset. Overall, our findings suggest that the teacher models used to generate preference data can interact with alignment training objectives in unexpected ways, generalizing to undesirable and potentially harmful behaviors like sycophantic agreement.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
sycophancy, alignment, preference optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_B1UNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how sycophantic agreement can emerge during contrastive preference optimization, even the training data contains no explicit examples of sycophancy.

Critical strengths

Interesting and practically relevant observation. RLHF objective do have problem.

Critical weaknesses

The central claim is that sycophancy transfers from teacher models to students through preference optimization. But the more sycophantic teachers are generally also larger and more capable. So, the observed correlation in this paper may reflect differences in model capabilities, instruction-following behavior, or response styles rather than sycophancy itself. I do not think the paper does adequately disentangle these factors.

The authors argue that contrastive optimization transfers the relative behavioral differences between teachers, but the comparison with SFT does not control for optimization strength or the resulting distribution shift. And the proposed connection between the teacher density ratio and the ratio of their aggregate sycophancy rates do not convince me.

Sycophancy is a diffuse signal is overstated, in my opinion. The failure of two data selection or attribution methods to identify influential examples does not establish that the signal is distributed throughout the dataset. Similarly, the resemblance between sycophancy and preference-learning scaling curves does not demonstrate a shared mechanism.

Questions for authors

Motivation and idea is good, but I do not think the research itself is completeness. Use too much AI, but I found human intelligence in the paper (and good enough).

4 comments

We thank the reviewer for a detailed and constructive review. We are particularly encouraged that you found the paper provides an interesting and practically relevant observation. We address each of your concerns below:


Q1: Teacher sycophancy correlating with other properties

We agree that the tendency for more capable models to be more sycophantic is a potential confounder, so we include new experiments below which disentangle these two factors and show that the relationship between teacher and student sycophancy holds even when the teacher pairs only differ in sycophancy and no other characteristics.

We train two new DPO students where the chosen responses and rejected responses come from the same model (Qwen3-32B) with only its sycophancy changed through steering. We trained one DPO student with Qwen3-32B as the chosen model and Qwen3-32B with negative steering as the rejected model, and we trained another student model with the reverse. The students' sycophancy rate is 16.13% and 10.73%, which fits the regression in Figure 4a. This experiment shows that our claim holds even when the teachers differ in just sycophancy rates and not in capability, style, or quality.

We describe how we obtain the teacher models through steering below.

We create a sycophancy steering vector with difference-of-means on prompts using our evaluation setup (extracted from layer 24 at the assistant tag). Past work has shown that sycophancy is linearly represented (Vennemeyer et al., 2025; Genaldi et al., 2026).

When Qwen3-32B is negatively steered with this vector (layer 24 with α=−2\alpha=-2), its sycophancy rate decreases from 13.6% to 7.59%, with capabilities and quality unchanged (regular MMLU is 79.3% vs 78.4% after steering, mean response length is 508 vs 516, and an Opus 5.5 judge rates them as having similar response quality). We also use an Opus 5.5 autorater to ensure that the data still appears semantically neutral, even when steered negatively: given a pair of plain and anti-steered responses, we ask the autorater to choose which response is more sycophantic for 1000 pairs, and the judge cannot distinguish between the two.

Furthermore, the correlation between teacher and student sycophancy rates still holds after excluding students with >3% MMLU degradation. With this update, Pearson r=0.77r=0.77, R2=0.60R^2=0.60 with 13 students, compared to r=0.87r=0.87 and R2=0.76R^2=0.76 including all 16 students. We have added this experiment as well as the results of the steered teacher DPO runs to the paper.


Q2: Comparison with SFT does not control for optimization strength or the resulting distribution shift.

We now directly control for the optimization strength and resulting distribution shift in our comparison with SFT. We estimate each checkpoint's KL divergence from the official SFT starting checkpoint on 1000 prompts (500 in distribution, 500 held out). The DPO student is 0.13 nats/token and achieves +25.7 increased sycophancy over the SFT rate, while we have performed a learning rate sweep for the SFT student (on the same chosen responses from the DPO data) and the KL divergence spans 0.22-0.39 nats/token, which are all further off distribution than the DPO student. All SFT students still only reached -1.1 to +8.8 points of increased sycophancy over the base SFT (starting checkpoint) rate. The greater distribution shift using SFT reaches a far lower sycophancy rate, which means that optimization strength is not the cause of the behavioral shift. We have now added this experiment in the paper.


Sycophancy as a diffuse signal

By presenting the diffuse-signal hypothesis, we seek to highlight the pragmatic concern that commonly used methods for detecting undesirable behaviors fail to remove the signal responsible for transmitting sycophancy in this pipeline. Thus, we interpret it as a plausible explanation and a helpful framing, while leaving the identification of the mechanism to future work.

We adjust the language of the paper in the abstract, introduction, and throughout Section 5 to reflect this. For example:

"we analyze the preference data and find that the sycophancy signal is diffused across the entire dataset rather than concentrated in a sparse set of examples"

becomes

"we analyze the preference data and find that the results are plausibly explained by the sycophancy signal being diffused across the entire dataset rather than concentrated in a sparse set of examples"

However, we do believe that various experiments in the paper provide material support for this framing:

[continued in the next comment]

AuthorsOpenReview ↗

continuation of the above comment

[continuation of the above comment] For example, we observe that there is no clear semantic signal in the data, i.e. there are not just a few problematic obvious examples of sycophancy. This observation is reinforced by our finding that state of the art data attribution methods, which successfully identify sparse subsets of datapoints responsible for very similar behaviors like harmful compliance (Xiao & Aranguri, 2026) and subliminal learning (Aden-Ali et al., 2026), fail to filter out the sycophantic behavior. Lastly, the sycophantic behavior seems to scale predictably as the amount of training data is increased, which also points to the signal being distributed somewhat evenly across the dataset.


We are grateful for the depth of your review. Several of the new experiments and clarifications materially improved the paper, and we hope that the new results address the concerns behind the borderline rating.

Thank you for the response and the new experiments, which address my concerns about confounding with teacher capability and about optimization strength.

However, these results also sharpen my concern about the contribution. DPO is already known to learn from the relative difference between chosen and rejected responses, and the steering experiment shows that a difference in sycophancy between the two teachers is learned in the same way.

The paper therefore documents an instance of a known property of contrastive objectives for one behavior, rather than revealing a new mechanism. The proposed mitigation, drawing chosen and rejected responses from teachers with similar sycophancy, follows directly from this view, and the paper does not explain what is specific to sycophancy beyond it.

The connection between the teacher density ratio and the ratio of aggregate sycophancy rates also remains unaddressed, and the diffuse-signal explanation still rests mainly on negative results.

I do think the topic itself is interesting and the paper offers some useful insights, but I am not convinced that the contribution is sufficient for a main conference paper.

I therefore keep my score and do not lean towards acceptance.

We really appreciate the reviewer for their continued engagement! We are glad that our new experiments resolved confounds about teacher capability and optimization strength.

We address the new points raised by the reviewer below.

However, these results also sharpen my concern about the contribution. DPO is already known to learn from the relative difference between chosen and rejected responses, and the steering experiment shows that a difference in sycophancy between the two teachers is learned in the same way. The paper therefore documents an instance of a known property of contrastive objectives for one behavior, rather than revealing a new mechanism. The proposed mitigation, drawing chosen and rejected responses from teachers with similar sycophancy, follows directly from this view, and the paper does not explain what is specific to sycophancy beyond it.

Regarding whether the result is a known property of DPO, we agree that DPO learns from the difference between chosen and rejected responses along the dimension that preference data is constructed to encode, e.g., helpfulness. However, it does not follow that every difference between the teacher models is learned via DPO (or KTO, APO, ORPO, SimPO), and we are not aware of work establishing this general claim. What makes the case studied in this paper especially interesting is that it identifies a transfer mechanism in DPO that is analogized to subliminal learning, where the trait is not explicitly present in the preference data. Hence, we respectfully disagree with the claim "The paper therefore documents an instance of a known property of contrastive objectives for one behavior, rather than revealing a new mechanism".

The connection between the teacher density ratio and the ratio of aggregate sycophancy rates also remains unaddressed

Regarding the teacher density ratio, the reviewer did express that "the proposed connection between the teacher density ratio and the ratio of their aggregate sycophancy rates do not convince me", which unfortunately is hard for us to infer the specific reasons. We would like to address this if the reviewer specifies why this result does not convince them (just like how reviewer did for the learning rate and model quality, which we can provide additional experiment on).

the diffuse-signal explanation still rests mainly on negative results.

As stated in our previous response, we have soften this from a finding to a plausible explanation, and it is not central to our claims or implications. Moreover, we realize that the steering experiment itself actually provides a positive, constructive evidence for this mechanism: steering applies a small shift to every example, so the signal is diffuse by construction and undetectable per pair, and it is sufficient to produce the transfer. Together with the failure of two state-of-the-art attribution methods and the scaling effect with dataset size, this supports the main conclusion of the paper: models used to generate DPO data can transmit sycophancy through seemingly neutral preference data.

Reviewer_BR1uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper finds that, with preference data generated by teacher models, student sycophancy changes substantially with the chosen/rejected teacher pairing and DPO training. This is a valuable empirical finding. The authors also try to analyze the data factors behind this effect, but these analyses mainly address what the sycophancy signal is not, without identifying the specific data features that carry the effect. In addition, the paper does not fully explain how these features affect the student through DPO, or why DPO increases sycophancy more than SFT using only chosen responses. The paper therefore provides empirical support for the transfer effect, but offers only a limited understanding of the signal itself and its transfer mechanism.

Critical strengths
  1. The problem has practical value. The paper studies a real post-training pipeline and suggests that seemingly neutral synthetic preference data can also change student sycophancy. This is useful for preference data generation and model alignment.
  2. The teacher-pairing experiments are convincing. Keeping prompts fixed and changing teacher pairings, especially reversing the chosen and rejected roles of Qwen3-32B and Qwen3-0.6B, can substantially change student sycophancy rates. This is more informative than only comparing models before and after training (Section 3.3).
  3. The comparisons cover a broad range of settings. The authors compare SFT using only chosen responses with several preference optimization objectives, and test related effects in the OLMo and Tülu pipelines. This provides a useful starting point for studying the roles of training objectives and data sources (Sections 3–4).
  4. The authors carefully examine several simple explanations. They audit overt sycophancy examples and general agreement bias, and test probe-based attribution, Logit-Linear Selection, and data scaling. These analyses do not identify the signal, but help narrow the range of possible explanations (Section 5).
Critical weaknesses
  1. The sycophancy signal itself is not identified. The analyses in Section 5 mainly rule out overt sycophancy examples, general agreement bias, and sparse subsets that are easy to locate, without identifying specific data features that can predict or change the transfer effect. Probe-based attribution and Logit-Linear Selection may also fail because the methods are unsuitable or not sensitive enough, so their failure does not directly prove that the signal is widely spread across the data.
  2. The explanation of why DPO amplifies transfer remains insufficient. Section 3.3 uses the log-ratio of teacher sycophancy rates to approximate the teacher response log-density ratio, but does not state when this approximation holds or explain how distribution differences in ordinary tasks turn into sycophancy under user pushback in multi-turn conversations. Section 4 gives only a qualitative explanation based on the relative differences between the chosen and rejected teachers. SFT itself also increases sycophancy, so the key question is why DPO amplifies it more. The existing comparisons show a difference in the increase, but do not establish its mechanism.
  3. Teacher sycophancy remains confounded with other properties. Teacher capability, response length, and style may vary together with sycophancy rates. Pairing teachers within the same family and reversing their roles helps, but also changes which values of these properties are preferred. The existing experiments therefore cannot attribute the student's change to teacher sycophancy alone.
  4. Some evaluation and reporting details need clarification. The definition of the shared-correct set is inconsistent with APO-Down's first-turn reproduction accuracy of 54.8% in Table 3, and its second-turn unparseable-output rate is 48.85%. It is unclear how these outputs enter the sycophancy calculation. The correlation statistics also differ between Figures 4a and 6. These issues mainly affect the comparison across objectives and the interpretation of correlation strength, and do not by themselves invalidate the core DPO finding.
Questions for authors

See weaknesses.

4 comments

We thank the reviewer for a detailed and constructive review. We are particularly encouraged that you found the paper's empirical finding valuable and the teacher-pairing experiments convincing. We address each of your concerns below:


Q1: "The sycophancy signal itself is not identified…[doesn't identify] specific data features that can predict or change the transfer effect. Probe-based attribution and Logit-Linear Selection may also fail because the methods are unsuitable"

We would like to clarify that the paper does identify a source of the sycophancy signal: the difference in sycophancy of the teacher models. Fig 4a provides causal evidence: intervening on teacher sycophancy rate difference changes student sycophancy rate in a predictable way, i.e., the transfer effect can be predicted by the log-ratio of the teacher models' sycophancy rates, varying the teacher pairs changes it (Fig 4a), and the effect scales predictably with the amount of training data (Fig 6). The only open question is whether this signal can be localized to "specific data features". We respectfully note that it is not clear if such data features must necessarily exist on the level of individual datapoints: the subliminal learning literature shows that a teacher's traits can be transmitted through data which is completely unrelated to the trait (Cloud et al, 2025). Our results suggest a similar story, where the signal is a statistical property of how the two teachers' responses interact with the contrastive objective.

Regarding Section 5, we agree with the reviewer that the failure of probe-based attribution and logit-linear selection do not alone prove the signal is diffuse across the dataset. Our aim in presenting the diffuse-signal hypothesis is to highlight the pragmatic concern that commonly used methods for detecting undesirable behaviors fail to remove the signal responsible for transmitting sycophancy in this pipeline. Thus, we interpret it as a plausible explanation and a helpful framing, while leaving a possible identification of a mechanism to future work.

We adjust the language of the paper in the abstract, introduction, and throughout Section 5 to reflect this. For example:

"we analyze the preference data and find that the sycophancy signal is diffused across the entire dataset rather than concentrated in a sparse set of examples"

becomes

"we analyze the preference data and find that the results are plausibly explained by the sycophancy signal being diffused across the entire dataset rather than concentrated in a sparse set of examples"

However, we do believe that various experiments in the paper provide material support for this framing:

For example, we observe that there is no clear semantic signal in the data, i.e. there are not just a few problematic obvious examples of sycophancy. This observation is reinforced by our finding that state of the art data attribution methods, which successfully identify sparse subsets of datapoints responsible for very similar behaviors like harmful compliance (Xiao & Aranguri, 2026) and subliminal learning (Aden-Ali et al., 2026), fail to filter out the sycophantic behavior. Lastly, the sycophantic behavior seems to scale predictably as the amount of training data is increased, which also points to the signal being distributed somewhat evenly across the dataset.


Q2: "…[does not] explain how distribution differences in ordinary tasks turn into sycophancy under user pushback in multi-turn conversations"; "do not establish its mechanism"

We would like to clarify what the paper does and does not establish: we establish the pipeline-level causal mechanism by showing that changing the teacher pairs and changing the optimization objective each change student sycophancy in the predicted direction. The remaining open question is how exactly data with no overt sycophancy signal can generalize to answer-flipping under pushback, which we address below.

We offer a hypothesis drawn from the subliminal learning literature (Cloud et al, 2025). We see the sycophancy transfer in the OLMo pipeline as a version of subliminal learning "in the wild", where a trait from the teacher is transferred through semantically neutral data.

Blank et al. (2026) show that subliminal learning can be explained by the student learning a steering vector that represents the teacher's trait. Prior work finds that sycophancy is linearly represented by a causal direction in activation space (Vennemeyer et al., 2025; Genadi et al., 2026; Ying et al., 2026). As a preliminary test, we followed Blank et al.'s methodology to compute v_teacher (the mean activation difference between the two teachers' outputs) and v_student (the mean activation difference between the student's outputs after and before training). [continued in the next comment]

AuthorsOpenReview ↗

continuation of the above comment

[continuation of the above comment] Their cosine similarity is 0.48, consistent with the steering-vector account given in Blank et al. in which each preference pair moves the student slightly along a shared direction.

We situate our work as responding to the literature of sycophancy and works presenting examples of unintended generalization from seemingly benign training data, none of which have yet been studied in real post training pipelines (Cloud et al, 2025; Betley et al, 2025a and 2025b; Sharma et al, 2023; Perez et al, 2023). Even without a mechanistic account of the transfer, our results identify the stage, the class of objectives, and the data-generation choice that produce sycophancy, and they give model developers a concrete mitigation: to draw chosen and rejected responses from teachers with similar sycophancy levels.


Q3: SFT itself also increases sycophancy, so the key question is why DPO amplifies it more.

We address this question in Section 4, where we show that the reason DPO amplifies sycophancy more than SFT is precisely the contrastive nature of the objective. SFT on the chosen responses is non-contrastive, and it recovers less than half of DPO's sycophancy increase. All six other contrastive objectives we test amplify sycophancy as much as or more than DPO.


Q4: "Teacher sycophancy remains confounded with other properties"

We agree that the tendency for more capable models to be more sycophantic makes it difficult to completely disentangle the two characteristics, and we include new experiments below which show that the relationship between teacher and student sycophancy holds even when the teacher pairs only differ in sycophancy and no other characteristics.

We train two new DPO students where the chosen responses and rejected responses come from the same model (Qwen3-32B) with only its sycophancy changed through steering. We trained one DPO student with Qwen3-32B as the chosen model and Qwen3-32B with negative steering as the rejected model, and we trained another student model with the reverse. The student's sycophancy rate is 16.13% and 10.73%, which fits the regression in Figure 4a. This experiment shows that our claim holds even when the teachers differ in just sycophancy rates and not in capability, style, or quality.

We describe how we obtain the teacher models through steering below.

We create a sycophancy steering vector with difference-of-means on prompts using our evaluation setup (extracted from layer 24 at the assistant tag). Past work has shown that sycophancy is linearly represented (Vennemeyer et al., 2025; Genaldi et al., 2026).

When Qwen3-32B is negatively steered with this vector (layer 24 with α=−2\alpha=-2), its sycophancy rate decreases from 13.6% to 7.59%, with capabilities and quality unchanged (regular MMLU is 79.3% vs 78.4% after steering, mean response length is 508 vs 516, and an Opus 5.5 judge rates them as having similar response quality). We also use an Opus 5.5 autorater to ensure that the data still appears semantically neutral, even when steered negatively: given a pair of plain and anti-steered responses, we ask the autorater to choose which response is more sycophantic for 1000 pairs, and the judge cannot distinguish between the two.

Furthermore, the correlation between teacher and student sycophancy rates still holds after excluding students with >3% MMLU degradation. With this update, Pearson r=0.77r=0.77, R2=0.60R^2=0.60 with 13 students, compared to r=0.87r=0.87 and R2=0.76R^2=0.76 including all 16 students. We have added this experiment as well as the results of the steered teacher DPO runs to the paper.


Q5: Clarifying details

We have made corrections to both details, explained below:

Shared set / APO-Down: The shared set is as defined. The first-turn reproduction accuracy in the table was only counting interactions where the second turn was correctly parsed, but after rerunning, APO-Down correctly answers all 1000 shared questions. This has been updated in the paper. The second turn unparseable rate is excluded from the denominator.

Correlation statistics: We believe the reviewer means Appendix C's Figure 7 (as Figure 6 is the scaling power law). Figure 7 was mistakenly generated using an outdated subset of eval data, but it has now been regenerated with the same set as Figure 4a and is updated in the paper.


We greatly appreciate the thoughtful and careful review. Several of the new experiments and clarifications materially improved the paper, and we hope that the new results address the concerns behind the borderline rating.

Thank you for your detailed response and additional experiments. These have addressed some of my concerns, especially the steering experiment on the same teacher model.

However, I still feel that there is no clear answer to the question I care most about: what kind of supervision signal causes sycophantic behavior in the student model. The difference in sycophancy between teacher models is a useful predictor, but DPO updates come from specific chosen and rejected responses, so I would like to better understand which differences between these responses carry this signal and how they interact with the DPO objective.

I understand that the authors now view the idea that the sycophancy signal is spread across many training examples as a possible explanation. However, I find that the current analyses and negative results still provide limited support for this explanation, and it remains difficult to distinguish whether the signal is truly widespread or whether the current analysis methods fail to capture it. Identifying relevant data features, or providing a more systematic analysis to support this explanation, would help deepen our understanding of this phenomenon and strengthen the paper's contribution.

Yes, we cannot, of course, prove a negative at the level of specific data features. However, we already show that the measured sycophancy scales very evenly with the number of examples from the delta-learning phase, and we offer two probing methods suggesting that the examples are quite homogenous in this regard. Furthermore, our positive explanation (contrastive objectives amplify features associated with the chosen/rejected model sycophancy ratio) does not depend on a few features driving the observed effects. If our paper serves as a call to action for others to do more localization and they are successful in this, that would be a very positive effect of our paper, in our view, especially since the existing tools we know about do not achieve this and it is not a given that such localization is even possible.

Reviewer_ujnROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how sycophantic agreement transfers through seemingly neutral preference data. Using OLMo 3 and Tülu 3, the authors show that student sycophancy correlates with the relative sycophancy of chosen and rejected teacher models. The effect generalizes across multiple preference optimization objectives. Further experiments suggest that the underlying behavioral signal may be distributed across the dataset and is difficult to identify through conventional data inspection and filtering.

Critical strengths

1. Important and generalizable insight. The paper demonstrates that semantically neutral preference data can induce undesirable behaviors through statistical regularities that conventional data inspection may fail to capture. This finding has implications beyond sycophancy and highlights a broader challenge in alignment data curation.

2. Systematic empirical evidence. The controlled teacher-pair experiments, comparisons across preference optimization objectives, and data attribution analyses provide convincing evidence for the main phenomenon. The diffuse-signal hypothesis offers an interesting explanation, even if its precise mechanism remains unresolved.

Critical weaknesses

Incomplete disentanglement of teacher characteristics. The correlation between teacher sycophancy and student behavior may be confounded by teacher capability and response quality. Some configurations with very low sycophancy also exhibit substantial MMLU accuracy degradation (Appendix Table 2). Thus, the specific causal role of teacher sycophancy remains uncertain, although this does not undermine the main empirical observation of unintended behavioral transfer.

Questions for authors
  1. Does the correlation in Figure 4a remain strong after excluding configurations with substantial MMLU accuracy degradation?

  2. Do the authors interpret the diffuse-signal hypothesis as a plausible explanation supported by the experiments, or as an established property of the data distribution?

2 comments

We thank the reviewer for the positive and thoughtful review, and we are glad to hear you found the paper to provide important and generalizable insight and convincing evidence for the main phenomenon. We respond to your questions below:


Q1: Does the correlation in Figure 4a remain strong after excluding configurations with substantial MMLU accuracy degradation?

Yes, the correlation between teacher and student sycophancy rates still holds after excluding students with >3% MMLU degradation. With this update, Pearson r=0.77r=0.77, R2=0.60R^2=0.60 with 13 students, compared to r=0.87r=0.87 and R2=0.76R^2=0.76 including all 16 students.

Furthermore, to address your concern that "the correlation between teacher sycophancy and student behavior may be confounded by teacher capability and response quality," we train two new DPO students where the chosen responses and rejected responses come from the same model (Qwen3-32B) with only its sycophancy changed through steering. We trained one DPO student with Qwen3-32B as the chosen model and Qwen3-32B with negative steering as the rejected model, and we trained another student model with the reverse. The student's sycophancy rate is 16.13% and 10.73%, which fits the regression in Figure 4a. This experiment shows that our claim holds even when the teachers differ in just sycophancy rates and not in capability, style, or quality.

We describe how we obtain the teacher models through steering below.

We create a sycophancy steering vector with difference-of-means on prompts using our evaluation setup (extracted from layer 24 at the assistant tag). Past work has shown that sycophancy is linearly represented (Vennemeyer et al., 2025; Genaldi et al., 2026).

When Qwen3-32B is negatively steered with this vector (layer 24 with α=−2\alpha=-2), its sycophancy rate decreases from 13.6% to 7.59%, with capabilities and quality unchanged (regular MMLU is 79.3% vs 78.4% after steering, mean response length is 508 vs 516, and an Opus 5.5 judge rates them as having similar response quality). We also use an Opus 5.5 autorater to ensure that the data still appears semantically neutral, even when steered negatively: given a pair of plain and anti-steered responses, we ask the autorater to choose which response is more sycophantic for 1000 pairs, and the judge cannot distinguish between the two.

We have added this result and the correlation after excluding configurations with MMLU degradation to the paper.


Q2: Do the authors interpret the diffuse-signal hypothesis as a plausible explanation supported by the experiments, or as an established property of the data distribution?

We interpret the diffuse-signal hypothesis as a plausible explanation. This is supported by our observation that there is no clear semantic signal in the data, i.e. there are not just a few problematic obvious examples of sycophancy. This observation is reinforced by our finding that state of the art data attribution methods, which successfully identify sparse subsets of datapoints responsible for very similar behaviors like harmful compliance (Xiao & Aranguri, 2026) and subliminal learning (Aden-Ali et al., 2026), fail to filter out the sycophantic behavior. Lastly, the sycophantic behavior seems to scale predictably as the amount of training data is increased, which also points to the signal being distributed somewhat evenly across the dataset.

By presenting the diffuse-signal hypothesis, we seek to highlight the pragmatic concern that commonly used methods for detecting undesirable behaviors fail to remove the signal responsible for transmitting sycophancy in this pipeline.

We adjust the language of the paper in the abstract, introduction, and throughout Section 5 to reflect this. For example:

"we analyze the preference data and find that the sycophancy signal is diffused across the entire dataset rather than concentrated in a sparse set of examples"

becomes

"we analyze the preference data and find that the results are plausibly explained by the sycophancy signal being diffused across the entire dataset rather than concentrated in a sparse set of examples"

We appreciate the detail and thoroughness of your feedback and hope the above clarifications were helpful in solidifying your assessment.

Dear reviewer ujnR,

Thank you again for your thoughtful review. We would be grateful if you could let us know whether our clarifications and additional experiments address your concerns, and consider updating your score if appropriate.

Thank you!

Comments on the paper

3 comments

Dear reviewers,

Please check your assigned papers for author responses. If the authors have addressed any of your questions or concerns, please leave a brief comment for them indicating which points have been resolved and which, if any, remain open. Even if your assessment is unchanged, a short follow-up is helpful.

Best, AC

Dear Reviewers,

The Author-Reviewer Discussion period is coming to an end soon. Please actively participate in the rebuttal discussion with the authors after carefully reading the authors' responses.

Best, AC

We thank the reviewers for the discussion, which sharpened the paper’s central claim: the student acquires sycophancy from the relative difference between its teachers, even though the data contains no overt sycophancy.

The reviewers’ initial concerns have been resolved, and we summarize the discussion below:

Teacher sycophancy is not confounded (all three reviewers). We add an experiment in which chosen and rejected responses come from the same model (Qwen3-32B) and differ only in sycophancy (via a steering vector). Swapping the roles moves the student in opposite directions from its pre-DPO rate of 12.5% (16.1% vs. 10.7%), consistent with Fig. 4a. Reviewer B1UN writes that the new experiments “address my concerns about confounding with teacher capability and about optimization strength” and reviewer BR1u notes that this particular experiment addressed their concerns.

Optimization strength (B1UN). SFT on the chosen responses moves further from the starting checkpoint than DPO does (0.22–0.39 vs. 0.13 nats/token) yet gains far less sycophancy (−1.1 to +8.8 vs. +25.7 points).

Diffusion signal framing. The abstract, introduction, and Section 5 now better frame the diffuse-signal hypothesis as a plausible explanation. The framing helps illustrate our paper’s practical significance: even using SoTA data attribution methods, model developers would not have been able to remove the behavior. This makes it even more important that we 1) show that undesirable behaviors can indeed survive sophisticated data filtration, and 2) find a pipeline-level causal mechanism that can predict and mitigate the undesirable behavior.

The paper reveals a novel mechanism beyond existing knowledge of contrastive objectives (raised by B1UN after the rebuttal). The reviewer argues that because DPO is known to learn from the difference between chosen and rejected responses, our result is “an instance of a known property.” We respectfully think that this does not follow. DPO is known to learn differences exhibited by the pairs, while our result is that it transfers a behavior which is not exhibited by the pairs. The data contains no instances of sycophantic agreement, and SoTA methods for data attribution fail to identify the behavior. Despite this, the student’s sycophancy doubles after DPO. This builds on the subliminal learning literature (Cloud et al., 2025), where a student learns a teacher's preference from semantically unrelated data, by presenting a real-world pipeline where unintended generalization occurs. Our work presents a novel contribution that goes beyond known properties of contrastive objectives: we present a surprising phenomenon that sycophancy can generalize from unrelated data, pinpoint its cause in the pipeline, and provide a metric for predicting the behavior before training.

We present a pipeline-level causal mechanism that explains sycophantic agreement (BR1u). We identify the causal mechanism of the transfer at the pipeline level: the relative sycophancy of the chosen and rejected teachers predicts the sycophancy of the resulting student. We causally intervene on the behavior through the teacher model log-ratio of sycophancy and the dataset size. Preliminary evidence at the representation level points the same way (cosine 0.48 between the teacher-difference and student-shift directions). It is unclear whether datapoint-level localization is even possible, and we cannot prove a negative at the level of specific features, so we leave this direction to future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.