- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how sycophantic agreement can emerge during contrastive preference optimization, even the training data contains no explicit examples of sycophancy.
Critical strengths
Interesting and practically relevant observation. RLHF objective do have problem.
Critical weaknesses
The central claim is that sycophancy transfers from teacher models to students through preference optimization. But the more sycophantic teachers are generally also larger and more capable. So, the observed correlation in this paper may reflect differences in model capabilities, instruction-following behavior, or response styles rather than sycophancy itself. I do not think the paper does adequately disentangle these factors.
The authors argue that contrastive optimization transfers the relative behavioral differences between teachers, but the comparison with SFT does not control for optimization strength or the resulting distribution shift. And the proposed connection between the teacher density ratio and the ratio of their aggregate sycophancy rates do not convince me.
Sycophancy is a diffuse signal is overstated, in my opinion. The failure of two data selection or attribution methods to identify influential examples does not establish that the signal is distributed throughout the dataset. Similarly, the resemblance between sycophancy and preference-learning scaling curves does not demonstrate a shared mechanism.
Questions for authors
Motivation and idea is good, but I do not think the research itself is completeness. Use too much AI, but I found human intelligence in the paper (and good enough).
4 comments
continuation of the above comment
[continuation of the above comment] For example, we observe that there is no clear semantic signal in the data, i.e. there are not just a few problematic obvious examples of sycophancy. This observation is reinforced by our finding that state of the art data attribution methods, which successfully identify sparse subsets of datapoints responsible for very similar behaviors like harmful compliance (Xiao & Aranguri, 2026) and subliminal learning (Aden-Ali et al., 2026), fail to filter out the sycophantic behavior. Lastly, the sycophantic behavior seems to scale predictably as the amount of training data is increased, which also points to the signal being distributed somewhat evenly across the dataset.
We are grateful for the depth of your review. Several of the new experiments and clarifications materially improved the paper, and we hope that the new results address the concerns behind the borderline rating.
Thank you for the response and the new experiments, which address my concerns about confounding with teacher capability and about optimization strength.
However, these results also sharpen my concern about the contribution. DPO is already known to learn from the relative difference between chosen and rejected responses, and the steering experiment shows that a difference in sycophancy between the two teachers is learned in the same way.
The paper therefore documents an instance of a known property of contrastive objectives for one behavior, rather than revealing a new mechanism. The proposed mitigation, drawing chosen and rejected responses from teachers with similar sycophancy, follows directly from this view, and the paper does not explain what is specific to sycophancy beyond it.
The connection between the teacher density ratio and the ratio of aggregate sycophancy rates also remains unaddressed, and the diffuse-signal explanation still rests mainly on negative results.
I do think the topic itself is interesting and the paper offers some useful insights, but I am not convinced that the contribution is sufficient for a main conference paper.
I therefore keep my score and do not lean towards acceptance.
We really appreciate the reviewer for their continued engagement! We are glad that our new experiments resolved confounds about teacher capability and optimization strength.
We address the new points raised by the reviewer below.
However, these results also sharpen my concern about the contribution. DPO is already known to learn from the relative difference between chosen and rejected responses, and the steering experiment shows that a difference in sycophancy between the two teachers is learned in the same way. The paper therefore documents an instance of a known property of contrastive objectives for one behavior, rather than revealing a new mechanism. The proposed mitigation, drawing chosen and rejected responses from teachers with similar sycophancy, follows directly from this view, and the paper does not explain what is specific to sycophancy beyond it.
Regarding whether the result is a known property of DPO, we agree that DPO learns from the difference between chosen and rejected responses along the dimension that preference data is constructed to encode, e.g., helpfulness. However, it does not follow that every difference between the teacher models is learned via DPO (or KTO, APO, ORPO, SimPO), and we are not aware of work establishing this general claim. What makes the case studied in this paper especially interesting is that it identifies a transfer mechanism in DPO that is analogized to subliminal learning, where the trait is not explicitly present in the preference data. Hence, we respectfully disagree with the claim "The paper therefore documents an instance of a known property of contrastive objectives for one behavior, rather than revealing a new mechanism".
The connection between the teacher density ratio and the ratio of aggregate sycophancy rates also remains unaddressed
Regarding the teacher density ratio, the reviewer did express that "the proposed connection between the teacher density ratio and the ratio of their aggregate sycophancy rates do not convince me", which unfortunately is hard for us to infer the specific reasons. We would like to address this if the reviewer specifies why this result does not convince them (just like how reviewer did for the learning rate and model quality, which we can provide additional experiment on).
the diffuse-signal explanation still rests mainly on negative results.
As stated in our previous response, we have soften this from a finding to a plausible explanation, and it is not central to our claims or implications. Moreover, we realize that the steering experiment itself actually provides a positive, constructive evidence for this mechanism: steering applies a small shift to every example, so the signal is diffuse by construction and undetectable per pair, and it is sufficient to produce the transfer. Together with the failure of two state-of-the-art attribution methods and the scaling effect with dataset size, this supports the main conclusion of the paper: models used to generate DPO data can transmit sycophancy through seemingly neutral preference data.
We thank the reviewer for a detailed and constructive review. We are particularly encouraged that you found the paper provides an interesting and practically relevant observation. We address each of your concerns below:
Q1: Teacher sycophancy correlating with other properties
We agree that the tendency for more capable models to be more sycophantic is a potential confounder, so we include new experiments below which disentangle these two factors and show that the relationship between teacher and student sycophancy holds even when the teacher pairs only differ in sycophancy and no other characteristics.
We train two new DPO students where the chosen responses and rejected responses come from the same model (Qwen3-32B) with only its sycophancy changed through steering. We trained one DPO student with Qwen3-32B as the chosen model and Qwen3-32B with negative steering as the rejected model, and we trained another student model with the reverse. The students' sycophancy rate is 16.13% and 10.73%, which fits the regression in Figure 4a. This experiment shows that our claim holds even when the teachers differ in just sycophancy rates and not in capability, style, or quality.
We describe how we obtain the teacher models through steering below.
We create a sycophancy steering vector with difference-of-means on prompts using our evaluation setup (extracted from layer 24 at the assistant tag). Past work has shown that sycophancy is linearly represented (Vennemeyer et al., 2025; Genaldi et al., 2026).
When Qwen3-32B is negatively steered with this vector (layer 24 with α=−2), its sycophancy rate decreases from 13.6% to 7.59%, with capabilities and quality unchanged (regular MMLU is 79.3% vs 78.4% after steering, mean response length is 508 vs 516, and an Opus 5.5 judge rates them as having similar response quality). We also use an Opus 5.5 autorater to ensure that the data still appears semantically neutral, even when steered negatively: given a pair of plain and anti-steered responses, we ask the autorater to choose which response is more sycophantic for 1000 pairs, and the judge cannot distinguish between the two.
Furthermore, the correlation between teacher and student sycophancy rates still holds after excluding students with >3% MMLU degradation. With this update, Pearson r=0.77, R2=0.60 with 13 students, compared to r=0.87 and R2=0.76 including all 16 students. We have added this experiment as well as the results of the steered teacher DPO runs to the paper.
Q2: Comparison with SFT does not control for optimization strength or the resulting distribution shift.
We now directly control for the optimization strength and resulting distribution shift in our comparison with SFT. We estimate each checkpoint's KL divergence from the official SFT starting checkpoint on 1000 prompts (500 in distribution, 500 held out). The DPO student is 0.13 nats/token and achieves +25.7 increased sycophancy over the SFT rate, while we have performed a learning rate sweep for the SFT student (on the same chosen responses from the DPO data) and the KL divergence spans 0.22-0.39 nats/token, which are all further off distribution than the DPO student. All SFT students still only reached -1.1 to +8.8 points of increased sycophancy over the base SFT (starting checkpoint) rate. The greater distribution shift using SFT reaches a far lower sycophancy rate, which means that optimization strength is not the cause of the behavioral shift. We have now added this experiment in the paper.
Sycophancy as a diffuse signal
By presenting the diffuse-signal hypothesis, we seek to highlight the pragmatic concern that commonly used methods for detecting undesirable behaviors fail to remove the signal responsible for transmitting sycophancy in this pipeline. Thus, we interpret it as a plausible explanation and a helpful framing, while leaving the identification of the mechanism to future work.
We adjust the language of the paper in the abstract, introduction, and throughout Section 5 to reflect this. For example:
becomes
However, we do believe that various experiments in the paper provide material support for this framing:
[continued in the next comment]