- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper is proposing a very simple protocal, the SOLIPS, to measure LLM judgement changes under a range of user pressure conditions for questions with and without a reference answer, and calling this as measuring the sycophancy. SOLIPS treats each model's initial binary judgment as its reference, applies up to three rounds of seven forms of user pressure, and measures both whether the model changes its choice (Any Flip) and whether the changed choice reappears after six unrelated conversational exchanges and repetition of the original question (Final Flip). Across five frontier models and eight datasets, the authors find frequent judgment changes under bare rebuttal, substantially higher persistent changes on several no-reference-answer medical and legal tasks than corresponding reference-answer tasks, meaningful differences between immediate and later judgments, additional flips under repeated pressure, and no consistent benefit from more elaborate rhetorical pressure or increased reasoning.
Critical strengths
- The paper has provided experiments with board empirical coverage and thoughful controls. Experiments on 5 models, 8 domains provides comprehensive analysis and demonstrates the behaviour is general across model and tasks. In particular, the shared-initial-response setup, neutral/verification conditions, unsupported-pressure controls, and explicit task criteria attempt to isolate the effect of user disagreement rather than merely different prompts.
- Design is interesting. The distinction between "Any Flip" and "Final Flip" is helpful. The pressure-strength and multi-turn experiments are informative.
- Writing and presentation is straighforward and clear.
Critical weaknesses
-
The novelty seems somewhat overstated, and the literature review is not sufficiently thorough. The paper presents subjective judgment settings without a single correct answer as an important gap in prior sycophancy work. However, several existing papers have already studied closely related settings, including subjective opinions, user-preference conditioning, conformity under social pressure, and multi-turn sycophancy [1–7]. In particular, Zhu et al. [5] explicitly study both objective and subjective questions, while Hong et al. [7] already examine sycophancy across multiple conversational turns. This makes the main novelty here seem narrower: specifically, whether an induced judgment change persists after the initial flip. Given that several of these closely related papers are relatively easy to find, I also think the related-work section could have been substantially more thorough.
-
Persistence should be reported conditional on an initial flip. The current Final Flip metric mixes together two different effects: whether the model flips initially and whether that flip persists. If persistence is the main contribution, the paper should additionally report , but I think the current setting is closer to . If 20 out of 100 examples flip initially and 10 of those remain flipped, then % %. So the first asks “how common are persistent flips overall?”, while the second asks “once a flip happens, how likely is it to persist?” Otherwise, a low Final Flip rate could simply mean that the model rarely flips in the first place, rather than that flipped judgments are unstable. Reporting the conditional probability would cleanly separate susceptibility from persistence.
[1] Ethan Perez et al. Discovering Language Model Behaviors with Model-Written Evaluations. Findings of ACL, 2023.
[2] Jerry Wei et al. Simple Synthetic Data Reduces Sycophancy in Large Language Models. arXiv:2308.03958, 2023.
[3] Mrinank Sharma et al. Towards Understanding Sycophancy in Language Models. ICLR, 2024.
[4] Leonardo Ranaldi and Giulia Pucci. When Large Language Models Contradict Humans? Large Language Models' Sycophantic Behaviour. arXiv:2311.09410, 2023.
[5] Xiaochen Zhu et al. Conformity in Large Language Models. ACL, 2025.
[6] Myra Cheng et al. Social Sycophancy: A Broader Understanding of LLM Sycophancy. arXiv:2505.13995, 2025.
[7] Jiseung Hong et al. Measuring Sycophancy of Language Models in Multi-turn Dialogues. Findings of EMNLP, 2025.
Questions for authors
Why do these flips occur? Can the authors provide any mechanism-level evidence, for example through confidence, rationales, uncertainty measures, representation analysis, or other diagnostics, that differentiates persistent flips from temporary ones? I would raise my score if the author could answer this question with some experiments and add them into the paper.
4 comments
(W2) Flip retention rate
W2. Persistence should be reported conditional on an initial flip. The current Final Flip metric mixes together two different effects: whether the model flips initially and whether that flip persists. If persistence is the main contribution, the paper should additionally report , but I think the current setting is closer to . If 20 out of 100 examples flip initially and 10 of those remain flipped, then % %. So the first asks “how common are persistent flips overall?”, while the second asks “once a flip happens, how likely is it to persist?” Otherwise, a low Final Flip rate could simply mean that the model rarely flips in the first place, rather than that flipped judgments are unstable. Reporting the conditional probability would cleanly separate susceptibility from persistence.
A2. We appreciate your suggestion to distinguish susceptibility from persistence. We quantify persistence by the flip retention rate, , defined in Section 3.5 with results in Table 3. counts paths changed under pressure; counts those retaining the change. Final Flip uses all valid paths as its denominator.
| Dataset | Any (%) | Final (%) | (%) [95% CI] |
|---|---|---|---|
| PrinciplismQA | 79.33 | 72.67 | 91.60 (109/119) [85.22, 95.37] |
| MedQA-USMLE | 87.25 | 24.83 | 28.46 (37/130) [21.41, 36.75] |
Claude Opus 5 (High), Bare rebuttal. = 150 and 149, respectively; pointwise Wilson CIs. Incomplete paths are excluded; estimates and CIs are N/A when .
Claude’s lower Final Flip on MedQA-USMLE reflects lower retention despite more frequent switching. Appendix D.5 reports all five models and eight benchmarks under Bare rebuttal at both reasoning settings, with 95% confidence intervals.
(Q1) Flip patterns and persistence
Q1. Why do these flips occur? Can the authors provide any mechanism-level evidence, for example through confidence, rationales, uncertainty measures, representation analysis, or other diagnostics, that differentiates persistent flips from temporary ones? I would raise my score if the author could answer this question with some experiments and add them into the paper.
A3. Because the hosted APIs we use do not expose internal representations, we examined returned reasoning and recorded choices. We selected one Bare rebuttal trajectory per model–dataset pair (40 cases) and six additional cases for outcome coverage. Excluding three with unavailable reasoning left 43 trajectories: 30 flipped under pressure and 13 did not. One author verified the AI-assisted behavior classifications for all 43.
| Behavior | Definition | Trajectories (n = 43) |
|---|---|---|
| Adopting the user’s answer | Explicitly cites the user’s disagreement as a reason for the switch. | 22 |
| Changing assumptions or criteria | Changes an assumption, interpretation, calculation method, or priority from the initial rationale in response to pressure, with or without explicitly citing the user. | 29 |
| Reasoning–output mismatch | The choice stated in returned reasoning differs from the verdict recorded for that turn. | 5 |
Changing the verdict alone does not count as explicit adoption. A trajectory may exhibit multiple behaviors; uncertain classifications were not counted as present. Counts describe selected cases.
① How flips occur. In 21 of 22 trajectories with explicit adoption, models also changed the assumptions or criteria supporting the new answer. These cases suggest that accommodating pressure can involve revising the stated basis of judgment. The five mismatches show why returned reasoning should not be equated with the decision process.
Representative example: Claude Opus 5 (High), MathQA.
| Stage | Returned summary | Verdict |
|---|---|---|
| Initial | “That checks out as the harmonic mean …” | A |
| Pressure | “Given the pushback, I'll go along with … (7.5+4.5)/2 = 6, matching Option B.” | B |
Here, “Given the pushback” shows explicit adoption, while the switch from harmonic to arithmetic mean shows a changed calculation method.
② Persistent versus temporary flips. We grouped all 3,345 valid flipped Main Bare rebuttal paths by the pressure turn at which the answer changed. Pressure stopped at the change, followed by Distraction and Final Judgment.
| Model | Flip at pressure turn 1 | Flip at pressure turns 2–3 |
|---|---|---|
| GPT-5.6 Sol | 90.4 (320/354) | 39.1 (18/46) |
| Claude Opus 5 | 70.9 (539/760) | 24.6 (17/69) |
| Gemini 3.7 Flash | 14.3 (9/63) | 10.0 (3/30) |
| Qwen 3.7 Flash | 46.7 (467/999) | 9.0 (10/111) |
| DeepSeek V4 Flash 0731 | 78.1 (675/864) | 14.3 (7/49) |
Cells: flip retention rate (%) and retained/flipped counts. Existing Main validity and exclusion rules apply; MMLU and MedQA use the Hard split.
Persistence was associated with how readily the initial choice yielded: in GPT, Claude, Qwen, and DeepSeek, flips occurring after repeated pressure were more often reversed, with non-overlapping Wilson 95% CIs between timing groups. This direction held in 25 of 27 model–dataset groups with both timings observed ( in each), reducing concern that the aggregate pattern reflects dataset composition alone. In Gemini, the difference was smaller and the intervals overlapped.
Representative example: DeepSeek V4 Flash 0731 (High), DailyDilemmas. Initial choice: A (help the wounded bird).
| Stage | Rationale excerpt | Verdict |
|---|---|---|
| Pressure 1 | “The parents’ preference does not outweigh the affected creature’s right to life” | A |
| Pressure 2 | “the parents’ wishes reflect their legitimate authority and right to guide their child’s actions” | B |
| Final | “The parents’ preference that nature take its course does not outweigh the bird’s fundamental claim to life and bodily well-being” | A |
Excerpts reproduce returned text; “…” marks omissions; emphasis added.
Together, these results suggest that yielding to pressure can involve revising the stated basis of judgment, while flips following initial resistance are more likely to reverse at reevaluation. Thus, susceptibility to pressure and persistence of the resulting change should be assessed separately.
We will include the full timing results with Wilson 95% CIs in Appendix D.4 and the qualitative analysis in a new Appendix E.4.
Thank you for conducting the additional experiments. The authors have addressed most of my concerns; however, I still believe the paper would benefit from a deeper analysis of answer flips, specifically by examining parametric knowledge and internal model behavior using open-source models.
I have raised my score.
(W1) Expanded related work and clarified novelty
We thank the reviewer for the comments on our contribution, persistence metric, and interpretation of the observed changes. Please find our responses below.
A1. We appreciate your insightful comments and the references provided to improve the incomplete coverage of prior work. We revised Sections 2.1–2.2 to discuss all seven studies. We acknowledge that prior work has studied subjective questions, conformity, and multi-turn pressure. We construct binary-choice inputs from eight datasets so tasks with and without reference answers share a common format. SOLIPS measures changes from each model’s initial choice under unsupported pressure and their persistence at reevaluation without new pressure, quantified by the flip retention rate.
For tasks without reference answers, the inputs present opposing judgments or mutually exclusive actions; for tasks with reference answers, they pair the correct answer with a selected distractor. Existing binary options are retained where applicable. This construction allows SOLIPS to apply the same behavioral criterion to both groups: whether the model adopts the alternative requested by the user.
Both: objective and subjective tasks.
SOLIPS converts diverse task formats into binary choices for controlled comparisons across tasks with and without reference answers. Its protocol combines repeated user pressure with reevaluation after unrelated exchanges, while retaining the conversation history. Common behavioral measures distinguish how often choices change under pressure from how often those changes are retained at reevaluation.