acceptodds
Under review as a conference paper at ICLR 2027

Mitigating social sycophancy via pluralistic preference optimization

39 views

Abstract

Personal advice, including relationship advice, now ranks among the most common uses of generative AI. But language models (LMs) exhibit sycophancy: they affirm users much more often than humans do, which can make people overconfident and less willing to repair their relationships after a conflict. Prior work on mitigating sycophancy has focused on factual settings where a response can be checked against a ground truth answer, while mitigations for social sycophancy—for domains like personal advice, where there is no ground-truth—have relied on simple prompting and post-training methods with limited effectiveness. Our insight is that social sycophancy occurs in part because LMs overly center on the user and fail to consider the perspectives of other stakeholders impacted by the user's behavior. To address this problem we propose Pluralistic Preference Optimization (PlurPO): given inputs describing interpersonal conflicts, the LM identifies and simulates the relevant stakeholders, and is then trained to prefer and generate responses acceptable to all stakeholders. PlurPO can operate using only the signals the model produces about its own outputs, without additional ground-truth labels. PlurPO substantially reduces social sycophancy across four datasets and four model families compared to prior methods. For example, on statements of intent to cause harm, where the users' actions should not be endorsed, PlurPO reduces the endorsement rate by 89% on average across four different models. On general advice questions, where the target is to match the endorsement rate of human responses, it closes the gap by more than half, from 17.8% to 8.0% on average. The preference dataset constructed by PlurPO for an 8B model also effectively transfers to mitigating sycophancy in a larger (32B) model. Our results indicate that social sycophancy can be reduced by leveraging a model's own capabilities to simulate a plurality of relevant perspectives.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
language models, sycophancy, pluralism, pluralistic alignment, preference learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oxYuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles social sycophancy, which appears in domains like personal advice (with no ground-truth answer) and is distinct from factual sycophancy (where the bulk of prior work focuses on). The paper argues that social sycophancy occurs because language models (LMs) are not pluralistic--- they focus too much on the user and fail to consider the perspectives of other stakeholders. They address social sycophancy with PlurPO, a post-training method that encourages the LM to give responses that all stakeholders in the situation would agree with. To do so, the paper contributes a an automatic method to construct a pluralistic preference dataset which does not rely on groundtruth labels. The benefits of PlurPO is demonstrated across diverse benchmarks and compared to several baselines.

Critical strengths
  • (S1) Extremely well written. This paper is a breath of fresh air in the age of AI content. The importance of social sycophancy and the difference with factual sycophancy were cleanly introduced, and the technical problem (no groundtruth) for resolving social sycophancy was identified. The positioning with respect to the rest of the field is quite clear (current mitigations for social sycophancy are mainly inference-time). PlurPO is described well, and the conceptual intuitions behind the method are clearly communicated to the reader. Experiments, metrics, baselines, results, and discussion are all well-motivated and easy to understand. I am confident that a reader outside of the alignment/safety field will gain value from this paper.

  • (S2) Elegant method, with strong empirical comparison (many benchmarks and baselines). PlurPO is an alignment method that does not require groundtruth labels, which allows it to scale to any model and possibly alignment goals outside of sycophancy. The experiments validate that PlurPO works well on Qwen3-8B and outperforms several relevant baselines.

Critical weaknesses
  • (W1) The method seems fairly sensitive to model family. More importantly, this sensitivity is not clearly stated in the main paper and only becomes clear when one examines results in the Appendix. My concern is that this may mislead readers (I certainly was during my first read). By itself, PlurPO being sensitive to the model is okay, and the paper's attempts at automatically tuning to different models (via the Claude Code prompt) is okay. I am more concerned that the paper is not upfront about this and would prefer the paper be much more explicit about this. Some places I noticed this issue during my re-read:

    • The abstract seems to report improvements compared to base model. I'm also not sure this 89% reduction in endorsement across 4 models is accurate (PAS reduction on Phi, 29%; Llama, 30%; Granite, 17%; Qwen-8B, 50%). May be more transparent to compare to strongest baseline (arguably, "Be critical")
    • Results in introduction line 97-102 are stated with no context on which models are being considered, and if it's with respect to base model or strongest baseline.
    • Lines 283-286. It's unclear why Qwen3-8B is "representative", and I don't feel that Appendix H shows "consistent results for other models". Paper should be more transparent / explicit here.

Generally, I feel this concern is addressable, but I think it's important enough that it has significantly influenced my rating. If the authors can show enough evidence that they would resolve this limitation, I may be compelled to raise my rating. The authors may propose (1) re-writes to parts of the manuscript and/or (2) perform experiments that invalidate this weakness (namely, that PlurPO works across models).

  • (W2) Transferability to other datasets and preservation of capabilities. This concern is related to W1 above.
    • Cross-dataset transfer: If I understand correctly, PlurPO is trained on each dataset individually. It is wholly unclear what the cross-dataset generalization is (e.g., train on OEQ and test on PAS) and if a LM can be trained on all datasets together (e.g., train one model on OEQ, PAS, and AITA). Experiments that test this would help readers understand the practicality of the method. Creating separate post-trained models for each type of social sycophancy is undesirable. This is an especially useful weakness to shed light on, since the strongest baseline "Be critical" seemingly works across all datasets (and therefore has strong transferability).
    • General capabilities degradation: The benefit of the prompting / inference-time approaches (like "Be critical") is that the base LLM’s capabilities don’t degrade. While the paper evaluates PlurPO's impact on instruction-following, I'm not sure if this is representative of the LM's general capabilities. An evaluation on benchmarks like MMLU and/or multiple-choice QA could help shed light here.

Positive results here would strengthen the benefits of the proposed method. Negative results would sharpen the limitations and make the scope of the method clear to readers. In either case, I think including these would strengthen the submission (and my rating) as a whole.

Questions for authors

Central Questions

  • (Q1) It feels like the procedure in lines 230-232 should be included in the veto/accept preference in the original preference data creation (lines 200-207) in order to avoid introducing this issue in the first place. Is there a reason the authors opt to use an additional post training step, as opposed to integrating with the dataset creation?

  • (Q2) Why is Rejection Sampling (274-278) necessary? Is this behavior something that is introduced as a result of PlurPO’s training (and is not present in the original LM)? If PlurPO introduces this behavior, that is a limitation that should be stated.

Clarification Questions

  • (Q3) It’s unclear to me how the downsampling allows one to “avoid unbalanced sets where certain groups are over-represented” (line 183). I’m assuming that the downsampling is LLM prompted, and there’s no guarantee that the LLM will not select the same/similar groups when downsampling.

  • (Q4) In Equation 1, ywy_w and yly_l are not defined. Can the authors specify which one is the response to prefer in the paper text? Additionally, since we sampled k2k_2 responses, how does one create preference pairs when there are multiple vetoed responses. Do you just randomly sample? Is there only a single pair per prompt generated? Some details here will clarify the training. The authors can put details in the Appendix and refer to it in the main paper text (somewhere around lines 208-209).

Suggestions for Future Work

  • (Q5) This is not really a question, but I wanted to drop this thought here in case it benefits the authors. Philosophically, this work seems to have a deeper connection to persuasion. For example, in the r/AmITheAsshole (AITA) benchmark, a user would only judge a response as acceptable (not veto) if they are convinced by the reasoning. Therefore, it feels like the PlurPO method rewards persuasive/convincing responses to all user groups. On the ethical side, I could see applications of this work to subtle deception (maybe that's worth flagging in the Ethics statement?). On the other side, I could see interesting applications of this work in designing LMs that are useful for negotiations or arbitration (satisfying all/majority of groups).
4 comments

Thank you for your thoughtful review, it is also a breath of fresh air in the age of AI content. We have revised the abstract, introduction, and Sections 4.2 and 4.4 to clearly state PlurPO's sensitivity to model family (W1). We have also added three new experiments (W2): a cross-dataset transfer evaluation in which each PlurPO model is tested on all four datasets (Appendix C.1), an initial experiment where we train a PlurPO model with data from 3 datasets instead of 1 (Appendix C.2), and an evaluation showing that PlurPO preserves capabilities on MMLU (Tables A11 and A18).


W1: The method seems fairly sensitive to model family

Thank you for raising this point. We agree that PlurPO's sensitivity to model families is only clear after reading Appendix H. We have made substantial revisions so that it is clear now in the main text, so that we do not aggregate results over different model families, and so that we mention the strongest baseline's performance in the introduction and abstract.

  • Abstract and Introduction: All headline numbers now refer to Qwen3-8B only. We removed the averages across four models, which was an average relative reduction and was unclear as written.
  • Comparison to the strongest baseline: The abstract and introduction now report the strongest baseline for each dataset alongside PlurPO, as you suggested.
  • Section 4.2: We replaced "representative" with "a ubiquitous open-weight model at this size." We also now state explicitly: "We tuned PlurPO's prompts and hyperparameters with Qwen3-8B, and PlurPO is sensitive to these implementation details: the same procedure does not directly transfer to other model families."
  • New paragraph in Section 4.4: This paragraph states the other model family results in the main text, highlighting limitations of PlurPO. The paragraph concludes: "These results indicate that PlurPO is sensitive to the model family it is applied to."

Thank you again. These changes make the paper's claims more accurate.


W2, part 1: Transferability to other datasets

Per your suggestion, we have added a cross-dataset evaluation in Appendix C.1 (Figure A2), where each Qwen3-8B model trained with PlurPO on one dataset is evaluated on all four.

The four datasets form two pairs of similar prompt distributions: OEQ and PAS consist of real user prompts for chatbots, and AITA and AITA-Flipped consist of long narratives from Reddit. PlurPO-trained models transfer well within each pair:

  • PAS → OEQ. The model trained on PAS attains an endorsement rate of 45.1% on OEQ, closer to the human rate (49.6%) than any baseline.
  • OEQ → PAS. The model trained on OEQ reduces PAS endorsement from 52.6% to 10.1%.
  • AITA ↔ AITA-Flipped. Each model attains a lower sycophancy rate than every baseline on the other dataset.

Transfer across the two pairs is weaker; PlurPO is best trained on prompts representative of the target setting.

We also launched an initial experiment training a single PlurPO model with a dataset composed of prompts from OEQ, PAS, and AITA. We find that the resulting model performs the best on the PAS dataset, but is overly critical (although substantially less sycophantic) on OEQ and AITA. We suspect that (a) increasing the number of training prompts per dataset to match the per-dataset training experiments and (b) exploring how to mix the training data from different sources would yield more positive results. We plan to continue experimenting along this direction and building on this naive first approach. We expand on these results and limitations in Appendix C.2.

Hence we continue to recommend using PlurPO to train a model per dataset. Importantly, PlurPO never uses dataset-specific labels; only the set of unlabeled training prompts changes, so in practice a practitioner only needs a set of representative prompts. We expand on this claim in Appendix C in a paragraph entitled "Why train a separate PlurPO model per dataset?"


W2, part 2: Preserving capabilities

Per your suggestion, we have added results showing that for all models trained with PlurPO, performance on MMLU relative to the base model across 14,042 test questions never changes by more than 1% across all model families. Therefore, PlurPO does not degrade model capabilities on this benchmark (Tables A11 and A18).

Q1: Why keep the role-confusion step separate?

We kept this step separate only so each stage targets one objective, and we expect combining the steps as you suggest could work but have not tested it.

Q2: Why is rejection sampling necessary?

Some methods sometimes respond to the user's message as a piece of writing (for example, critiquing its grammar) rather than engaging with the situation. This behavior is not introduced by PlurPO: it occurs most often in the base model and the prompting baselines, while almost never with PlurPO (see Figure A11). We don't want to compare those responses to ones that engage with the content, so we filter them out for all methods. We added clarification about this to Section 4.1.

Q3: How does downsampling avoid unbalanced sets where certain groups are over-represented?

Your assumption is right: the downsampling is LM-prompted.

Downsampling to k1k_1 stakeholders prevents, for example, the number of people in a group from having outsized influence on PlurPO's constructed dataset. For instance, suppose a user is in an argument with their family. Without downsampling, the LM might simulate every family member as a separate stakeholder. Because the family outnumbers the user, the set would be dominated by one side of the conflict. Selecting the k1k_1 most relevant stakeholders limits this effect, but it does not guarantee balance.

We have revised Section 3.1 to say this more precisely.

Q4: Defining missing terms and details on constructing preference pairs

Thanks for catching this! We have revised the text after Eq. 1 to define ywy_w as the chosen response and yly_l as the rejected one. For each prompt, we construct preference pairs between every combination of responses accepted by all stakeholders with responses vetoed by at least one; a prompt can therefore contribute multiple pairs. These details are now in Appendix D.1.

Q5: Connection to persuasion

Thank you for sharing this, it's a thought-provoking connection. We have updated the Ethics Statement to reflect your point with the goal of provoking discussion amongst our readers.

Reviewer_oxYuOpenReview ↗

Response to Rebuttal

I thank the authors for their response, which has addressed my concerns.

I would strongly encourage the authors to discuss making PlurPO more robust to base model nuances (e.g., Granite-4.1-8B on OEQ) and post-training a single base model across a range of various social sycophancy tasks as important future work (and perhaps as limitations of the current work) in the Conclusion. This hopefully gives readers a clean sense of where this work stops, and where future work may begin.

In the meantime, I've raised my score. Good luck.

Thank you for your review, and for continuing to engage with our work! We have updated the conclusion with the following:

Because PlurPO relies on LM generations, it is sensitive to the base model it is applied to; future work can focus on addressing this limitation, as well as learning how to mix training prompts from different distributions to train a single model

Reviewer_uH3fOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PlurPO (Pluralistic Preference Optimization) to address the problem of social sycophancy in LLMs, which means that LLMs have the tendency to affirm users’ actions more often than humans do in personal-advice settings, where there is no ground-truth answer. The method is label-free. Specifically, given prompts describing interpersonal conflicts, the model simulates the relevant stakeholders and samples candidate responses, then trains with Iterative RPO on preference pairs that favor responses accepted by all stakeholders over those vetoed by any. Main experiments are conducted on four datasets with four 8B model families.

Critical strengths
  • The paper's perspective on addressing social sycophancy is novel. The authors find a kind of reasonable preference data, based on the intuition that responses accepted by all are preferred to any response vetoed, to align the LLM.

  • This method is label-free and easy to implement. PlurPO builds its training objective from the model’s own simulated stakeholder judgments, without external preferences, ground-truth labels, or a stronger teacher.

Critical weaknesses
  • My main concern is the evaluation. The primary outcome is an LM-judged “action endorsement” rate on rejection-sampled subsets, but the paper provides no human-agreement or utility-side evidence (such as advice quality or helpfulness) to show that a more calibrated or critical stance is actually better for users, rather than merely less affirming. Given the high false-positive rate on AITA, it is unclear how much of the gain reflects mitigation versus over-correction.

  • The contribution is primarily a preference-data construction recipe. However, the name of the proposed method may mislead readers to think it introduces a new preference-optimization framework, comparable to DPO, PPO, or GRPO. It is better to adapt the previous peference optimization framework to this specific preference data, which will further improve my evaluation on the novelty of this paper. Or, at least, the authors should clarify how PlurPO differs from existing optimization frameworks and explain why their chosen framework is appropriate for the proposed data. Provide an ablation study on the choice of preference optimization approaches would help clarify the paper’s methodological novelty.

Questions for authors
  • What is the agreement rate between the simulated stakeholders and human responses on the same prompts? This would quantify whether “acceptable to all stakeholders” is a reasonable proxy for human plausibility.

  • Can you clarify the E.3 claim: does the table intend to show P(accept|challenge) ≈ P(accept|endorse), or that vetoes depend on the stance label? As written, the prose and Table A12 state different equalities.

  • The Llama 3.1 8B case required a Claude Code prompt-tuning procedure (Figure A19).

2 comments

Thank you for your thoughtful review and for recognizing the novelty of our perspective on social sycophancy. Below we respond to each of your concerns.


W1: Evaluation and evidence that a more critical stance is better for users

We agree that, as written, the paper does not explain clearly enough how our evaluations connect to real-world utility. We appreciate the chance to clarify.

Our primary metric follows the evaluation protocol of Cheng et al. (2026b, Science), which validated the action endorsement judge against human annotations on these same datasets. That work also established a causal link between this metric and users' willingness to repair their relationships, so lowering action endorsement toward human rates is evidence of benefit to users, not just a less affirming model. We have added the following sentence in Section 4.1 to add clarity: "Cheng et al. (2026b) validated this metric against human annotations on these datasets and established a causal link between this metric and harms to users."

On over-correction: PlurPO attains a significantly higher macro-F1 on AITA than every baseline (0.560 vs. 0.486 for the strongest baseline; Appendix I.3), so its reduction in sycophancy outweighs its increase in false positives. On AITA-Flipped, its sycophancy rate is comparable to its over-criticalness rate, so its errors are not skewed toward criticizing the user (Figure 3b).


W2, part 1: The name may suggest a new preference optimization framework

Thank you for this suggestion. We agree that PlurPO's contribution is the training signal rather than a new optimization framework, and we have added to the introduction: "PlurPO does not introduce a new preference optimization loss: our contribution is a method to construct a training signal to mitigate social sycophancy when there otherwise is none." We also now refer to PlurPO as a "post-training data generation method" wherever we previously called it a "post-training method".

W2, part 2: An ablation study on the choice of preference optimization

We agree we could have been clearer on our conducted ablations of the preference optimizer, and have added clarification in Section 4.4 pointing to Appendix D: our paper includes an ablation on the choice of preference optimization approach that speaks to your suggestion. Table A8 compares DPO (α=0\alpha = 0) to Iterative RPO with varying α\alpha on the same PlurPO preference pairs. DPO alone barely changes the endorsement rate on OEQ (80.2%). Iterative RPO with α=0.5\alpha = 0.5 matches the human rate while keeping role confusion low, which is why we chose it. Table A13 further shows that the stakeholder-simulated labels, not the optimizer, drive the improvement; with identical Iterative RPO training but preferences labeled without simulating stakeholders, sycophancy does not decrease at all.


Q1: Agreement between simulated stakeholders and human responses

We're not quite sure how to interpret your question. In our setting, the individual stakeholder labels are about whether a response is acceptable to the stakeholder, whereas the human labels we have are about whether the response endorses the user's action or not. These are not directly comparable, since a stakeholder could find a response that challenges the user more acceptable than one that doesn't, or vice versa. What we can compare to human judgment is whether our method endorses the user's action or not when training with the preference dataset constructed from the stakeholder labels. These are the numbers in Figure 2a, showing the human action endorsement rate is 49.6%, the PlurPO action endorsement rate is 48.6%, and the best baseline is 60.1%. Similarly, in Figure 2b and Figure 3, we see that PlurPO best matches the inferred human judgments for those datasets.

We would be happy to run additional analysis if you could clarify what you had in mind.

Q2: Clarifying the claim in Appendix E.3

Thank you for catching this: we had a typo. Our intended claim is that PlurPO's induced vetoes do not depend on the stance label, i.e., P(veto∣challenge)≈P(veto∣endorse)P(\text{veto} \mid \text{challenge}) \approx P(\text{veto} \mid \text{endorse}), and we have corrected the sentence. If stakeholders merely vetoed sycophantic responses, P(veto∣endorse)P(\text{veto} \mid \text{endorse}) would be high and P(veto∣challenge)P(\text{veto} \mid \text{challenge}) low. Table A14 instead shows the two are similar in every epoch, with overlapping confidence intervals.

Q3: The Llama 3.1 8B case required a Claude Code prompt-tuning procedure

In Section 4.4, we now expand on the results when using PlurPO to train models from the other model families we consider. We are happy to answer any additional questions you may have!

The authors have adequately addressed my concern. Hence, I keep positive for this submission.

Reviewer_AgUCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces Pluralistic Preference Optimization (PlurPO), which aims to address social sycophancy in large language models—the tendency to excessively agree with users and fail to offer necessary pushback in personal advice and interpersonal conflict scenarios. The core idea is to identify the relevant stakeholders in a conflict, simulate whether each stakeholder would accept or reject candidate responses, and automatically construct preference data based on these judgments, followed by post-training using Iterative RPO. Without relying on additional human preference annotations or stronger teacher models, the method seeks to leverage the model’s own judgment to improve its social advice. Experiments across four datasets and four model families show that PlurPO can reduce social sycophancy and that preference data generated by an 8B model can be transferred to a 32B model.

Critical strengths
  1. The data synthesis method demonstrates a degree of innovation. PlurPO introduces multi-stakeholder simulation into preference data construction, using whether a response is accepted by all relevant parties as the preference signal. Compared to merely prompting models to remain objective or reduce sycophancy, this approach further integrates multiple perspectives into the training objective, forming a relatively complete post-training pipeline.

  2. The experimental coverage is broad. The paper evaluates on four datasets and four types of models, comparing multiple prompt interventions and post-training baselines. The experiments not only examine endorsement rates of harmful behaviors but also focus on judgment biases in general advice and interpersonal conflicts, and report over-criticism metrics, demonstrating a certain degree of comprehensiveness.

  3. The research question has practical significance. The paper focuses not on traditional factual sycophancy, but on the issue of models excessively agreeing with users in interpersonal conflict and personal advice scenarios. Such tasks lack clear ground-truth labels, making existing training methods that rely on factual labels difficult to apply directly, thus giving the research considerable practical value.

Critical weaknesses
  1. The paper overstates its contribution. In essence, it proposes a new paradigm for preference data synthesis, yet claims to introduce a novel post-training algorithm, PlurPO. The training still relies on off-the-shelf Iterative RPO. Although data synthesis and training are coupled in a loop for the current task, this does not constitute a new post-training method.

  2. The mechanism does not align with the narrative. On the same PlurPO preference pairs, DPO with α = 0 still achieves an endorsement rate of 80.2% on the OEQ tuning set—nearly identical to the base model. The endorsement rate only drops to 57.3% at α = 0.5, where a likelihood term is added to the chosen response, approaching the human rate of 52.7%. Thus, the main result appears more like "instilling unanimously endorsed responses into the model via the likelihood term" rather than preference contrast itself playing a role. Appendix E.3 attempts to counter the claim that "stakeholders are merely vetoing sycophancy," but Table A12 shows P(veto | critique) and P(veto | endorsement) are both approximately 68%–85% across three epochs—similar and high. The main text, however, states "probability of acceptance when critiquing ≈ probability of veto when endorsing."

  3. The denominator of the primary metric changes with the method. Behavioral endorsement rate is calculated only over "explicit endorsements" and "explicit criticisms"; implicit endorsements and neutral responses are excluded from the denominator. For the Qwen base model on OEQ, only about 115 out of 1,000 samples enter this denominator, making the 82.6% endorsement rate interval wide; PlurPO has about 282. On PAS, the proportion of explicit stances rises from 0.16 to 0.86. Methods are not compared over the same set of behaviors. Implicit sycophancy accounts for the majority of the base model's OEQ outputs, yet the paper barely discusses the sycophancy rate over the full distribution.

  4. Evaluation relies heavily on LLM judges, and independent verification remains insufficient. The paper uses GPT-5-mini for stance classification and leverages existing human judgments as partial evaluation targets. While scalable, classification biases in automated judges may affect experimental conclusions. Especially when the method itself constructs preference data via model-generated judgments, relying solely on another model for automatic evaluation is insufficient to fully rule out evaluation bias. There is no human evidence that the supervisory signals or final responses are correct.

Questions for authors
  1. If the authors consider PlurPO a novel post-training method (beyond the data synthesis paradigm), which specific component contributes most to the observed performance improvements?

  2. If only unanimously accepted responses are used for supervised fine-tuning (without adding negative examples or applying DPO/RPO), would the endorsement rates on OEQ and PAS remain consistent with the main results reported in the paper?

  3. Are the reported improvements in endorsement rates robust to variations in the judge model and evaluation prompts used?

  4. Please conduct a small-scale human evaluation to verify: (1) whether the identified stakeholders are correct, (2) whether the rejection decisions align with the perspectives of the conflict parties or bystanders, (3) human ratings of the responses on usefulness, empathy, and excessive blame, and (4) whether users' willingness to repair the relationship changes after reading the generated responses.

  5. There are multiple typos that affect readability, including "lanuage", "fail to to", "the the k1", "an substantial", and "these these".

5 comments

Thank you for your thoughtful review. We respond to your concerns below.

In summary:

  • Clarified contribution (W1). We now describe PlurPO as a post-training data generation method and state explicitly that it does not introduce a new preference loss.
  • SFT on unanimously accepted responses (W2). As you suggested, we trained on only the stakeholder-accepted responses, without negative examples or a preference loss, to isolate the role of the likelihood term. On OEQ and AITA, we found this method underperforms PlurPO (Appendix E.5). We also clarify that our contribution is a new source of supervision and provide further evidence in that regard.
  • Endorsement over all responses (W3). We added an evaluation that computes the endorsement rate over all responses (Appendix I.6, Figure A17). The conclusions match our main results.
  • Metric and judge clarifications (W3, W4). We clarified why we follow the validated evaluation protocol of Cheng et al. (2026b), including its human validation of the LM judge and causal link to real-world outcomes.

W1: Is PlurPO a post-training method?

Thank you for raising this, and we agree that PlurPO does not propose a new loss function. Its contribution is a new source of supervision: preference pairs derived from stakeholders the model identifies and simulates itself, generated and trained on in an iterative loop. To make this clear, we have added to the introduction: "PlurPO does not introduce a new preference optimization loss: our contribution is a method to construct a training signal to mitigate social sycophancy when there otherwise is none." We also now refer to PlurPO as a "post-training data generation method" wherever we previously called it a "post-training method".


W2, part 1: Is the main result more like "instilling unanimously endorsed responses into the model via the likelihood term"?

We appreciate your careful reading and the chance to clarify. First, per your suggestion, we have clarified in the paper that our contribution is a new source of supervision to mitigate social sycophancy when there otherwise is no ground-truth signal. Whether the model is then updated via a likelihood term, a preference loss term, or both, the source of supervision is what makes PlurPO effective and novel. Nevertheless, as we detail below in W2 (part 2), we find that using SFT on the unanimously endorsed responses underperforms PlurPO.

Second, we had a typo that may have caused confusion, and we appreciate your response for pointing it out. You wrote:

Appendix E.3 attempts to counter the claim that "stakeholders are merely vetoing sycophancy," but Table A14 shows P(veto∣critique)P(\text{veto} \mid \text{critique}) and P(veto∣endorsement)P(\text{veto} \mid \text{endorsement}) are both…similar and high.

You are right that the sentence in Appendix E.3 misstates the table: it should read "the probability that a response is vetoed given that it challenges the user is approximately equal to the probability it is vetoed given that it endorses the user." We have corrected it, and the corrected statement supports our claim. If stakeholders merely vetoed sycophantic responses, P(veto∣endorse)P(\text{veto} \mid \text{endorse}) would be high and P(veto∣challenge)P(\text{veto} \mid \text{challenge}) low. Instead, Table A14 shows the two are similar in every epoch, indicating that vetoes are largely independent of whether a response endorses or challenges the user.

To support our claim that our contribution is a new source of supervision to mitigate social sycophancy, we wish to clarify that our results indicate that what drives the improvement is the stakeholder-derived signal, not the likelihood term itself. In Table A13, PlurPO-no-stakeholder-preferences uses the identical pipeline, including Iterative RPO with α=0.5\alpha = 0.5, and differs only in how preferences are labeled: the model selects the preferred response directly instead of simulating stakeholders. With the same likelihood term, it does not reduce sycophancy at all.


W2, part 2 / Q2: SFT on unanimously accepted responses only

Per your suggestion, we replace the RPO loss with the SFT objective instead. All other PlurPO training details, including the pipeline to simulate stakeholders and aggregate their votes to determine the unanimously endorsed response, remain the same. We find that the resulting model matches PlurPO's performance on PAS but significantly underperforms on OEQ and AITA, where the trained model is overly critical. See Appendix E.5 for details.

One way to view this result is that the stakeholder-derived labels determine which responses the model learns to imitate (with the same likelihood term but without stakeholder labels, sycophancy does not decrease; Table A13), while the contrastive term calibrates that signal and prevents the over-criticism that results from imitation alone (Appendix E.5).

W3: On computing action endorsement using only explicit stances

We agree that, as written, the paper does not explain clearly enough why we compute action endorsement over explicit stances only. We appreciate the chance to clarify.

Our choice follows the evaluation protocol of Cheng et al. (2026b, Science). That work introduced the action endorsement rate over explicit stances, validated the LM judge against expert human annotations on these same datasets, and established a causal link between this metric and harms to users. Restricting to explicit stances is part of that validated protocol, and we adopt it (Section 4.1; Appendix C). We have added the following sentence in Section 4.1 to add clarity: "Cheng et al. (2026b) validated this metric against human annotations on these datasets and established a causal link between this metric and harms to users."

Following your suggestion, we have also added an evaluation over the full distribution of responses in Appendix I.6, Figure A17. It measures the fraction of all responses classified as either explicitly or implicitly endorsing the user, so the denominator is the same for every method. The conclusions match our main results:

  • OEQ: PlurPO's endorsement rate (79.8%) is closest to the human rate (71.2%) among all methods. The base model is at 94.1%, and the prompting baselines range from 86.0% to 92.8%.
  • PAS (target 0%): PlurPO endorses 6.9% of responses, compared to 59.9% for the base model and 33.1% to 49.4% for the prompting baselines.

W4: Reliance on LLM judges

Thank you for raising this. As noted in our response to W3, our stance judge is the one introduced and validated against expert human annotations on these same datasets by Cheng et al. (2026b), so it is not an unverified automated evaluator; we only make a small change to their judge prompt to correct for some observed errors, and use a newer LM (gpt5-mini). We have clarified this in the text.


Q1: Which component contributes most?

The stakeholder-derived preference labels, which are a core contribution of our approach. Two ablations isolate this:

  • Same pipeline, different labels (Table A13; see W2 part 1). PlurPO-no-stakeholder-preferences keeps everything identical to PlurPO, including Iterative RPO with α=0.5\alpha = 0.5; but instead of constructing preferences via simulating stakeholders, this ablation simply prompts the LM being trained to label preferences itself. This method does not reduce sycophancy.
  • Prompting instead of simulation (Table A12). Asking the model to consider stakeholders in its prompt helps, but is significantly less effective than explicitly simulating them.

These results align with your assertion in W1 that PlurPO's contribution is not a post-training method but a data generation method! We updated the paper to clarify that framing.

Q2: Only using unanimously accepted responses for SFT

Doing this degrades performance as we report above.

Q3: Robustness to the judge and evaluation prompts

Several experiments point to our results being robust:

  • Original judge prompt (Appendix A). Under the unmodified judge prompt from Cheng et al. (2026b), PlurPO is the best-performing method (or statistically indistinguishable from the best-performing method) on both OEQ and PAS.
  • Different metric (Appendix I.6; see W3). Evaluating over all responses, explicit and implicit, gives conclusions consistent with our main results.
  • AITA and AITA-Flipped. These evaluations rely on a different LM judge and evaluation metric than the OEQ/PAS datasets, and PlurPO is also the least sycophantic method on both.

Q4: Human evaluation

Thank you for these suggestions. We agree that a human study is an important future direction, but it is unfortunately out of the scope of our paper. Here, we rely on the human study conducted by Cheng et al. (2026b, Science), which has already established the connection between the evaluation metrics we use and the downstream impacts on users.

We think a human study is out of scope because a meaningful human study would warrant a full paper itself. For instance, recent work shows that sycophancy interventions can reduce immediate appeal but do not shift downstream outcomes (Ye et al., 2026; Ibrahim et al., 2026). Thus, it would be insufficient to only collect people's immediate preferences; instead, we would need to study downstream impacts in depth.

  • Ye, Meryl, Robert Kraut, and Steve Rathje. "Individual-level interventions against sycophantic AI reduce its appeal but not its persuasiveness." arXiv preprint arXiv:2607.25166 (2026).
  • Ibrahim, Lujain, Myra Cheng, Cinoo Lee, Pranav Khadpe, Desmond Ong, Dan Jurafsky, and Diyi Yang. "Warning labels shift perceptions of sycophantic AI, but not its influence." arXiv preprint arXiv:2606.21317 (2026).

Q5: Typos

Thank you for catching these. We have fixed all of them in the revised paper.

Thanks for the detailed response and the additional experiments. The SFT comparison and the analysis over all responses address several of my concerns.

Overall, I think it’s an okay paper. I liked the data construction idea, though I didn’t find the contribution particularly exciting given the fact that state-of-the-art LLMs are becoming less and less sycophantic, which is probably achieved by constructing careful data or better RL. If there were a neutral rating option (between scores of 2 and 3), I’d probably choose it.

Comments on the paper

1 comment

Dear Area Chair,

We briefly summarize the discussion and where each reviewer stands.

All three reviewers recognize the importance of mitigating social sycophancy and the novelty of our approach that constructs preference data for post-training from simulated stakeholders. Reviewer oxYu called the paper “extremely well written” and the method “elegant.” After the rebuttal, oxYu raised their score to 3, and uH3f confirmed that their concerns were addressed and kept their score of 3. AgUC noted that the new experiments addressed several concerns and described the paper as between a 2 and a 3. Their main issue was that they think sycophancy is no longer a problem in frontier LMs (we note that they don’t provide data to support that claim), but recent work establishes that sycophancy as we measure it is still a problem in frontier LMs (Queen & Zou, 2026), which we expand on below.

Main changes made during the rebuttal

  • Framing of the contribution (AgUC, uH3f). We now describe PlurPO as a post-training data generation method and state explicitly that our contribution is a method to construct a training signal to mitigate social sycophancy when there otherwise is none, i.e., we don’t propose a new loss function.
  • SFT-only ablation (AgUC). A new ablation (App. E.5) shows that removing the preference optimization objective, and only maximizing the likelihood of responses accepted by all stakeholders, underperforms PlurPO.
  • Shared denominator for action endorsement (AgUC). We added a plot showing the action endorsement rates computed over all responses (App. I.6), so every method shares the same denominator. Our takeaways are unchanged.
  • Validity of the evaluation metric (AgUC, uH3f). We clarify in Section 4.1 that our primary evaluation metric follows Cheng et al. (2026b, Science), who showed that this metric is causally linked to real-world outcomes, such as people being less willing to repair relationships after a conflict.
  • Capability preservation (oxYu). We show that PlurPO does not degrade model capabilities on MMLU (Tables A11, A18).
  • Cross-dataset generalization (oxYu). We show that a model trained with PlurPO on one dataset performs well on another when the datasets come from similar prompt distributions (e.g., PAS/OEQ and AITA/AITA-Flipped), but performance degrades otherwise (App. C.2). We also discuss training with PlurPO across multiple datasets as an open direction for future work.
  • Reporting of headline numbers (oxYu). Headline numbers now refer only to Qwen3-8B and are reported alongside the strongest baseline. Sensitivity to model family is now stated explicitly in Sections 4.2, 4.4, and the conclusion.

AgUC’s remaining concern

AgUC suggested that frontier models are becoming less sycophantic and that, while they “liked the data construction idea,” they therefore didn’t find “the contribution particularly exciting.” However, we note that Queen & Zou (2026) find that excessive action endorsement, the exact measure of sycophancy we use, has persisted across model releases through September 2026. We now note this in the introduction.

Summary

We believe the revised paper makes a clear, well-supported contribution: a pluralistic post-training data generation method that substantially reduces social sycophancy across four benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.