- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tackles social sycophancy, which appears in domains like personal advice (with no ground-truth answer) and is distinct from factual sycophancy (where the bulk of prior work focuses on). The paper argues that social sycophancy occurs because language models (LMs) are not pluralistic--- they focus too much on the user and fail to consider the perspectives of other stakeholders. They address social sycophancy with PlurPO, a post-training method that encourages the LM to give responses that all stakeholders in the situation would agree with. To do so, the paper contributes a an automatic method to construct a pluralistic preference dataset which does not rely on groundtruth labels. The benefits of PlurPO is demonstrated across diverse benchmarks and compared to several baselines.
Critical strengths
-
(S1) Extremely well written. This paper is a breath of fresh air in the age of AI content. The importance of social sycophancy and the difference with factual sycophancy were cleanly introduced, and the technical problem (no groundtruth) for resolving social sycophancy was identified. The positioning with respect to the rest of the field is quite clear (current mitigations for social sycophancy are mainly inference-time). PlurPO is described well, and the conceptual intuitions behind the method are clearly communicated to the reader. Experiments, metrics, baselines, results, and discussion are all well-motivated and easy to understand. I am confident that a reader outside of the alignment/safety field will gain value from this paper.
-
(S2) Elegant method, with strong empirical comparison (many benchmarks and baselines). PlurPO is an alignment method that does not require groundtruth labels, which allows it to scale to any model and possibly alignment goals outside of sycophancy. The experiments validate that PlurPO works well on Qwen3-8B and outperforms several relevant baselines.
Critical weaknesses
-
(W1) The method seems fairly sensitive to model family. More importantly, this sensitivity is not clearly stated in the main paper and only becomes clear when one examines results in the Appendix. My concern is that this may mislead readers (I certainly was during my first read). By itself, PlurPO being sensitive to the model is okay, and the paper's attempts at automatically tuning to different models (via the Claude Code prompt) is okay. I am more concerned that the paper is not upfront about this and would prefer the paper be much more explicit about this. Some places I noticed this issue during my re-read:
- The abstract seems to report improvements compared to base model. I'm also not sure this 89% reduction in endorsement across 4 models is accurate (PAS reduction on Phi, 29%; Llama, 30%; Granite, 17%; Qwen-8B, 50%). May be more transparent to compare to strongest baseline (arguably, "Be critical")
- Results in introduction line 97-102 are stated with no context on which models are being considered, and if it's with respect to base model or strongest baseline.
- Lines 283-286. It's unclear why Qwen3-8B is "representative", and I don't feel that Appendix H shows "consistent results for other models". Paper should be more transparent / explicit here.
Generally, I feel this concern is addressable, but I think it's important enough that it has significantly influenced my rating. If the authors can show enough evidence that they would resolve this limitation, I may be compelled to raise my rating. The authors may propose (1) re-writes to parts of the manuscript and/or (2) perform experiments that invalidate this weakness (namely, that PlurPO works across models).
- (W2) Transferability to other datasets and preservation of capabilities. This concern is related to W1 above.
- Cross-dataset transfer: If I understand correctly, PlurPO is trained on each dataset individually. It is wholly unclear what the cross-dataset generalization is (e.g., train on OEQ and test on PAS) and if a LM can be trained on all datasets together (e.g., train one model on OEQ, PAS, and AITA). Experiments that test this would help readers understand the practicality of the method. Creating separate post-trained models for each type of social sycophancy is undesirable. This is an especially useful weakness to shed light on, since the strongest baseline "Be critical" seemingly works across all datasets (and therefore has strong transferability).
- General capabilities degradation: The benefit of the prompting / inference-time approaches (like "Be critical") is that the base LLM’s capabilities don’t degrade. While the paper evaluates PlurPO's impact on instruction-following, I'm not sure if this is representative of the LM's general capabilities. An evaluation on benchmarks like MMLU and/or multiple-choice QA could help shed light here.
Positive results here would strengthen the benefits of the proposed method. Negative results would sharpen the limitations and make the scope of the method clear to readers. In either case, I think including these would strengthen the submission (and my rating) as a whole.
Questions for authors
Central Questions
-
(Q1) It feels like the procedure in lines 230-232 should be included in the veto/accept preference in the original preference data creation (lines 200-207) in order to avoid introducing this issue in the first place. Is there a reason the authors opt to use an additional post training step, as opposed to integrating with the dataset creation?
-
(Q2) Why is Rejection Sampling (274-278) necessary? Is this behavior something that is introduced as a result of PlurPO’s training (and is not present in the original LM)? If PlurPO introduces this behavior, that is a limitation that should be stated.
Clarification Questions
-
(Q3) It’s unclear to me how the downsampling allows one to “avoid unbalanced sets where certain groups are over-represented” (line 183). I’m assuming that the downsampling is LLM prompted, and there’s no guarantee that the LLM will not select the same/similar groups when downsampling.
-
(Q4) In Equation 1, and are not defined. Can the authors specify which one is the response to prefer in the paper text? Additionally, since we sampled responses, how does one create preference pairs when there are multiple vetoed responses. Do you just randomly sample? Is there only a single pair per prompt generated? Some details here will clarify the training. The authors can put details in the Appendix and refer to it in the main paper text (somewhere around lines 208-209).
Suggestions for Future Work
- (Q5) This is not really a question, but I wanted to drop this thought here in case it benefits the authors. Philosophically, this work seems to have a deeper connection to persuasion. For example, in the r/AmITheAsshole (AITA) benchmark, a user would only judge a response as acceptable (not veto) if they are convinced by the reasoning. Therefore, it feels like the PlurPO method rewards persuasive/convincing responses to all user groups. On the ethical side, I could see applications of this work to subtle deception (maybe that's worth flagging in the Ethics statement?). On the other side, I could see interesting applications of this work in designing LMs that are useful for negotiations or arbitration (satisfying all/majority of groups).
4 comments
Q1: Why keep the role-confusion step separate?
We kept this step separate only so each stage targets one objective, and we expect combining the steps as you suggest could work but have not tested it.
Q2: Why is rejection sampling necessary?
Some methods sometimes respond to the user's message as a piece of writing (for example, critiquing its grammar) rather than engaging with the situation. This behavior is not introduced by PlurPO: it occurs most often in the base model and the prompting baselines, while almost never with PlurPO (see Figure A11). We don't want to compare those responses to ones that engage with the content, so we filter them out for all methods. We added clarification about this to Section 4.1.
Q3: How does downsampling avoid unbalanced sets where certain groups are over-represented?
Your assumption is right: the downsampling is LM-prompted.
Downsampling to stakeholders prevents, for example, the number of people in a group from having outsized influence on PlurPO's constructed dataset. For instance, suppose a user is in an argument with their family. Without downsampling, the LM might simulate every family member as a separate stakeholder. Because the family outnumbers the user, the set would be dominated by one side of the conflict. Selecting the most relevant stakeholders limits this effect, but it does not guarantee balance.
We have revised Section 3.1 to say this more precisely.
Q4: Defining missing terms and details on constructing preference pairs
Thanks for catching this! We have revised the text after Eq. 1 to define as the chosen response and as the rejected one. For each prompt, we construct preference pairs between every combination of responses accepted by all stakeholders with responses vetoed by at least one; a prompt can therefore contribute multiple pairs. These details are now in Appendix D.1.
Q5: Connection to persuasion
Thank you for sharing this, it's a thought-provoking connection. We have updated the Ethics Statement to reflect your point with the goal of provoking discussion amongst our readers.
Response to Rebuttal
I thank the authors for their response, which has addressed my concerns.
I would strongly encourage the authors to discuss making PlurPO more robust to base model nuances (e.g., Granite-4.1-8B on OEQ) and post-training a single base model across a range of various social sycophancy tasks as important future work (and perhaps as limitations of the current work) in the Conclusion. This hopefully gives readers a clean sense of where this work stops, and where future work may begin.
In the meantime, I've raised my score. Good luck.
Thank you for your review, and for continuing to engage with our work! We have updated the conclusion with the following:
Because PlurPO relies on LM generations, it is sensitive to the base model it is applied to; future work can focus on addressing this limitation, as well as learning how to mix training prompts from different distributions to train a single model
Thank you for your thoughtful review, it is also a breath of fresh air in the age of AI content. We have revised the abstract, introduction, and Sections 4.2 and 4.4 to clearly state PlurPO's sensitivity to model family (W1). We have also added three new experiments (W2): a cross-dataset transfer evaluation in which each PlurPO model is tested on all four datasets (Appendix C.1), an initial experiment where we train a PlurPO model with data from 3 datasets instead of 1 (Appendix C.2), and an evaluation showing that PlurPO preserves capabilities on MMLU (Tables A11 and A18).
W1: The method seems fairly sensitive to model family
Thank you for raising this point. We agree that PlurPO's sensitivity to model families is only clear after reading Appendix H. We have made substantial revisions so that it is clear now in the main text, so that we do not aggregate results over different model families, and so that we mention the strongest baseline's performance in the introduction and abstract.
Thank you again. These changes make the paper's claims more accurate.
W2, part 1: Transferability to other datasets
Per your suggestion, we have added a cross-dataset evaluation in Appendix C.1 (Figure A2), where each Qwen3-8B model trained with PlurPO on one dataset is evaluated on all four.
The four datasets form two pairs of similar prompt distributions: OEQ and PAS consist of real user prompts for chatbots, and AITA and AITA-Flipped consist of long narratives from Reddit. PlurPO-trained models transfer well within each pair:
Transfer across the two pairs is weaker; PlurPO is best trained on prompts representative of the target setting.
We also launched an initial experiment training a single PlurPO model with a dataset composed of prompts from OEQ, PAS, and AITA. We find that the resulting model performs the best on the PAS dataset, but is overly critical (although substantially less sycophantic) on OEQ and AITA. We suspect that (a) increasing the number of training prompts per dataset to match the per-dataset training experiments and (b) exploring how to mix the training data from different sources would yield more positive results. We plan to continue experimenting along this direction and building on this naive first approach. We expand on these results and limitations in Appendix C.2.
Hence we continue to recommend using PlurPO to train a model per dataset. Importantly, PlurPO never uses dataset-specific labels; only the set of unlabeled training prompts changes, so in practice a practitioner only needs a set of representative prompts. We expand on this claim in Appendix C in a paragraph entitled "Why train a separate PlurPO model per dataset?"
W2, part 2: Preserving capabilities
Per your suggestion, we have added results showing that for all models trained with PlurPO, performance on MMLU relative to the base model across 14,042 test questions never changes by more than 1% across all model families. Therefore, PlurPO does not degrade model capabilities on this benchmark (Tables A11 and A18).