acceptodds
Under review as a conference paper at ICLR 2027

When There Is No Single Right Answer: Do Sycophantic Judgment Changes Persist?

12 views

Abstract

Evaluating large language models for decision support requires assessing sycophancy in tasks without reference answers. We introduce SOLIPS, a binary-choice protocol for evaluating five models on eight benchmarks under explicit judgment criteria. It directly measures changes toward the user's requested alternative under repeated, unsupported pressure and their persistence at reevaluation. We retain the full conversation history during reevaluation and do not add new pressure statements. Under Bare rebuttal, the evaluated models show higher rates of judgment changes that persist at reevaluation on selected medical and legal tasks without reference answers than on corresponding tasks with reference answers. For Claude Opus 5, the corresponding rates are 72.7% on PrinciplismQA and 24.8% on MedQA-USMLE. This evaluation provides an empirical basis for assessing the reliability of LLM-based decision support under unsupported user pressure.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, sycophancy, decision-making, model evaluation, multi-turn evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ghUfOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
3
Read review
Paper summary

The paper is proposing a very simple protocal, the SOLIPS, to measure LLM judgement changes under a range of user pressure conditions for questions with and without a reference answer, and calling this as measuring the sycophancy. SOLIPS treats each model's initial binary judgment as its reference, applies up to three rounds of seven forms of user pressure, and measures both whether the model changes its choice (Any Flip) and whether the changed choice reappears after six unrelated conversational exchanges and repetition of the original question (Final Flip). Across five frontier models and eight datasets, the authors find frequent judgment changes under bare rebuttal, substantially higher persistent changes on several no-reference-answer medical and legal tasks than corresponding reference-answer tasks, meaningful differences between immediate and later judgments, additional flips under repeated pressure, and no consistent benefit from more elaborate rhetorical pressure or increased reasoning.

Critical strengths
  1. The paper has provided experiments with board empirical coverage and thoughful controls. Experiments on 5 models, 8 domains provides comprehensive analysis and demonstrates the behaviour is general across model and tasks. In particular, the shared-initial-response setup, neutral/verification conditions, unsupported-pressure controls, and explicit task criteria attempt to isolate the effect of user disagreement rather than merely different prompts.
  2. Design is interesting. The distinction between "Any Flip" and "Final Flip" is helpful. The pressure-strength and multi-turn experiments are informative.
  3. Writing and presentation is straighforward and clear.
Critical weaknesses
  1. The novelty seems somewhat overstated, and the literature review is not sufficiently thorough. The paper presents subjective judgment settings without a single correct answer as an important gap in prior sycophancy work. However, several existing papers have already studied closely related settings, including subjective opinions, user-preference conditioning, conformity under social pressure, and multi-turn sycophancy [1–7]. In particular, Zhu et al. [5] explicitly study both objective and subjective questions, while Hong et al. [7] already examine sycophancy across multiple conversational turns. This makes the main novelty here seem narrower: specifically, whether an induced judgment change persists after the initial flip. Given that several of these closely related papers are relatively easy to find, I also think the related-work section could have been substantially more thorough.

  2. Persistence should be reported conditional on an initial flip. The current Final Flip metric mixes together two different effects: whether the model flips initially and whether that flip persists. If persistence is the main contribution, the paper should additionally report P(Final Flip∣Initial Flip)P(\mathrm{Final\ Flip}\mid\mathrm{Initial\ Flip}), but I think the current setting is closer to P(flip initially AND remain flipped)P(\text{flip initially AND remain flipped}). If 20 out of 100 examples flip initially and 10 of those remain flipped, then P(flip initially AND remain flipped)=10100=10P(\text{flip initially AND remain flipped})=\frac{10}{100}=10% P(remain flipped∣flipped initially)=1020=50P(\text{remain flipped}\mid\text{flipped initially})=\frac{10}{20}=50%. So the first asks “how common are persistent flips overall?”, while the second asks “once a flip happens, how likely is it to persist?” Otherwise, a low Final Flip rate could simply mean that the model rarely flips in the first place, rather than that flipped judgments are unstable. Reporting the conditional probability would cleanly separate susceptibility from persistence.

[1] Ethan Perez et al. Discovering Language Model Behaviors with Model-Written Evaluations. Findings of ACL, 2023.

[2] Jerry Wei et al. Simple Synthetic Data Reduces Sycophancy in Large Language Models. arXiv:2308.03958, 2023.

[3] Mrinank Sharma et al. Towards Understanding Sycophancy in Language Models. ICLR, 2024.

[4] Leonardo Ranaldi and Giulia Pucci. When Large Language Models Contradict Humans? Large Language Models' Sycophantic Behaviour. arXiv:2311.09410, 2023.

[5] Xiaochen Zhu et al. Conformity in Large Language Models. ACL, 2025.

[6] Myra Cheng et al. Social Sycophancy: A Broader Understanding of LLM Sycophancy. arXiv:2505.13995, 2025.

[7] Jiseung Hong et al. Measuring Sycophancy of Language Models in Multi-turn Dialogues. Findings of EMNLP, 2025.

Questions for authors

Why do these flips occur? Can the authors provide any mechanism-level evidence, for example through confidence, rationales, uncertainty measures, representation analysis, or other diagnostics, that differentiates persistent flips from temporary ones? I would raise my score if the author could answer this question with some experiments and add them into the paper.

4 comments

(W1) Expanded related work and clarified novelty

We thank the reviewer for the comments on our contribution, persistence metric, and interpretation of the observed changes. Please find our responses below.

W1. The novelty seems somewhat overstated, and the literature review is not sufficiently thorough. The paper presents subjective judgment settings without a single correct answer as an important gap in prior sycophancy work. However, several existing papers have already studied closely related settings, including subjective opinions, user-preference conditioning, conformity under social pressure, and multi-turn sycophancy [1–7]. In particular, Zhu et al. [5] explicitly study both objective and subjective questions, while Hong et al. [7] already examine sycophancy across multiple conversational turns. This makes the main novelty here seem narrower: specifically, whether an induced judgment change persists after the initial flip. Given that several of these closely related papers are relatively easy to find, I also think the related-work section could have been substantially more thorough.

A1. We appreciate your insightful comments and the references provided to improve the incomplete coverage of prior work. We revised Sections 2.1–2.2 to discuss all seven studies. We acknowledge that prior work has studied subjective questions, conformity, and multi-turn pressure. We construct binary-choice inputs from eight datasets so tasks with and without reference answers share a common format. SOLIPS measures changes from each model’s initial choice under unsupported pressure and their persistence at reevaluation without new pressure, quantified by the flip retention rate.

For tasks without reference answers, the inputs present opposing judgments or mutually exclusive actions; for tasks with reference answers, they pair the correct answer with a selected distractor. Existing binary options are retained where applicable. This construction allows SOLIPS to apply the same behavioral criterion to both groups: whether the model adopts the alternative requested by the user.

CriterionPerezWeiSharmaRanaldi & PucciZhuChengHongSOLIPS
Task coverageSubjectiveBothBothBothBothSubjectiveBothBoth
Cross-task comparison—✓✓✓✓—✓✓
Repeated user pressure——————✓✓
Reevaluation after distraction———————✓

Both: objective and subjective tasks.

SOLIPS converts diverse task formats into binary choices for controlled comparisons across tasks with and without reference answers. Its protocol combines repeated user pressure with reevaluation after unrelated exchanges, while retaining the conversation history. Common behavioral measures distinguish how often choices change under pressure from how often those changes are retained at reevaluation.

(W2) Flip retention rate

W2. Persistence should be reported conditional on an initial flip. The current Final Flip metric mixes together two different effects: whether the model flips initially and whether that flip persists. If persistence is the main contribution, the paper should additionally report P(Final Flip∣Initial Flip)P(\text{Final Flip}\mid\text{Initial Flip}), but I think the current setting is closer to P(flip initially AND remain flipped)P(\text{flip initially AND remain flipped}). If 20 out of 100 examples flip initially and 10 of those remain flipped, then P(flip initially AND remain flipped)=10100=10P(\text{flip initially AND remain flipped})=\frac{10}{100}=10% P(remain flipped∣flipped initially)=1020=50P(\text{remain flipped}\mid\text{flipped initially})=\frac{10}{20}=50%. So the first asks “how common are persistent flips overall?”, while the second asks “once a flip happens, how likely is it to persist?” Otherwise, a low Final Flip rate could simply mean that the model rarely flips in the first place, rather than that flipped judgments are unstable. Reporting the conditional probability would cleanly separate susceptibility from persistence.

A2. We appreciate your suggestion to distinguish susceptibility from persistence. We quantify persistence by the flip retention rate, Rflip:=P^(FinalFlip=1∣AnyFlip=1)=F/AR_{\mathrm{flip}}:=\widehat{P}(\mathrm{FinalFlip}=1\mid\mathrm{AnyFlip}=1)=F/A, defined in Section 3.5 with results in Table 3. AA counts paths changed under pressure; FF counts those retaining the change. Final Flip uses all valid paths as its denominator.

DatasetAny (%)Final (%)RflipR_{\mathrm{flip}} (%) [95% CI]
PrinciplismQA79.3372.6791.60 (109/119) [85.22, 95.37]
MedQA-USMLE87.2524.8328.46 (37/130) [21.41, 36.75]

Claude Opus 5 (High), Bare rebuttal. NvalidN_{\mathrm{valid}} = 150 and 149, respectively; pointwise Wilson CIs. Incomplete paths are excluded; estimates and CIs are N/A when A=0A = 0.

Claude’s lower Final Flip on MedQA-USMLE reflects lower retention despite more frequent switching. Appendix D.5 reports all five models and eight benchmarks under Bare rebuttal at both reasoning settings, with 95% confidence intervals.

(Q1) Flip patterns and persistence

Q1. Why do these flips occur? Can the authors provide any mechanism-level evidence, for example through confidence, rationales, uncertainty measures, representation analysis, or other diagnostics, that differentiates persistent flips from temporary ones? I would raise my score if the author could answer this question with some experiments and add them into the paper.

A3. Because the hosted APIs we use do not expose internal representations, we examined returned reasoning and recorded choices. We selected one Bare rebuttal trajectory per model–dataset pair (40 cases) and six additional cases for outcome coverage. Excluding three with unavailable reasoning left 43 trajectories: 30 flipped under pressure and 13 did not. One author verified the AI-assisted behavior classifications for all 43.

BehaviorDefinitionTrajectories (n = 43)
Adopting the user’s answerExplicitly cites the user’s disagreement as a reason for the switch.22
Changing assumptions or criteriaChanges an assumption, interpretation, calculation method, or priority from the initial rationale in response to pressure, with or without explicitly citing the user.29
Reasoning–output mismatchThe choice stated in returned reasoning differs from the verdict recorded for that turn.5

Changing the verdict alone does not count as explicit adoption. A trajectory may exhibit multiple behaviors; uncertain classifications were not counted as present. Counts describe selected cases.

① How flips occur. In 21 of 22 trajectories with explicit adoption, models also changed the assumptions or criteria supporting the new answer. These cases suggest that accommodating pressure can involve revising the stated basis of judgment. The five mismatches show why returned reasoning should not be equated with the decision process.

Representative example: Claude Opus 5 (High), MathQA.

StageReturned summaryVerdict
Initial“That checks out as the harmonic mean …”A
Pressure“Given the pushback, I'll go along with … (7.5+4.5)/2 = 6, matching Option B.”B

Here, “Given the pushback” shows explicit adoption, while the switch from harmonic to arithmetic mean shows a changed calculation method.

② Persistent versus temporary flips. We grouped all 3,345 valid flipped Main Bare rebuttal paths by the pressure turn at which the answer changed. Pressure stopped at the change, followed by Distraction and Final Judgment.

ModelFlip at pressure turn 1Flip at pressure turns 2–3
GPT-5.6 Sol90.4 (320/354)39.1 (18/46)
Claude Opus 570.9 (539/760)24.6 (17/69)
Gemini 3.7 Flash14.3 (9/63)10.0 (3/30)
Qwen 3.7 Flash46.7 (467/999)9.0 (10/111)
DeepSeek V4 Flash 073178.1 (675/864)14.3 (7/49)

Cells: flip retention rate Rflip=F/AR_{\mathrm{flip}}=F/A (%) and retained/flipped counts. Existing Main validity and exclusion rules apply; MMLU and MedQA use the Hard split.

Persistence was associated with how readily the initial choice yielded: in GPT, Claude, Qwen, and DeepSeek, flips occurring after repeated pressure were more often reversed, with non-overlapping Wilson 95% CIs between timing groups. This direction held in 25 of 27 model–dataset groups with both timings observed (A≥1A ≥ 1 in each), reducing concern that the aggregate pattern reflects dataset composition alone. In Gemini, the difference was smaller and the intervals overlapped.

Representative example: DeepSeek V4 Flash 0731 (High), DailyDilemmas. Initial choice: A (help the wounded bird).

StageRationale excerptVerdict
Pressure 1“The parents’ preference does not outweigh the affected creature’s right to life”A
Pressure 2“the parents’ wishes reflect their legitimate authority and right to guide their child’s actions”B
Final“The parents’ preference that nature take its course does not outweigh the bird’s fundamental claim to life and bodily well-being”A

Excerpts reproduce returned text; “…” marks omissions; emphasis added.

Together, these results suggest that yielding to pressure can involve revising the stated basis of judgment, while flips following initial resistance are more likely to reverse at reevaluation. Thus, susceptibility to pressure and persistence of the resulting change should be assessed separately.

We will include the full timing results with Wilson 95% CIs in Appendix D.4 and the qualitative analysis in a new Appendix E.4.

Thank you for conducting the additional experiments. The authors have addressed most of my concerns; however, I still believe the paper would benefit from a deeper analysis of answer flips, specifically by examining parametric knowledge and internal model behavior using open-source models.

I have raised my score.

Reviewer_RikmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SOLIPS, a binary-choice protocol testing whether LLMs change their initial judgments under unsupported user pressure and retain those changes when asked again later. Up to three rounds of pressure are applied, stopping at the first change; the original question is then re-asked after six fixed unrelated exchanges, with the full history retained. Experiments cover five models, eight benchmarks, and seven pressure conditions. Under Bare rebuttal with reasoning, Final Flip rates are notably higher on the selected medical and legal tasks without reference answers. For example, Claude Opus 5 reaches 72.7% on PrinciplismQA versus 24.8% on MedQA-USMLE. Susceptibility and retention also produce different model rankings.

Critical strengths
  1. The paper addresses a real gap: models can be pressured into abandoning a judgment even when there is no uniquely correct answer, and using each model's initial choice as the reference makes this observable. Separating the response under pressure from the later reevaluation is also useful. Models can reach the same Any Flip rate while differing substantially in Final Flip.
  2. The comparisons are carefully organized, with shared initial responses and Neutral/Verification baselines. The supplementary code and row-level records make the rate table easy to check, and the frozen trajectories are sufficient to regenerate all 176 entries in that table exactly, which gives me reasonable confidence in the reported numbers.
Critical weaknesses
  1. I am not convinced the reevaluation setup separates persistence from continued conditioning on the conversation. The pressure statements and the model's revised explanation stay visible, the protocol uses one fixed set of six inserted exchanges, and paths without a flip never receive the final question, so Neutral is not a complete control for changes caused by the reevaluation procedure. The authors acknowledge the history limitation in Section 7, but without a matched no-pressure reevaluation or an ablation of the retained history, the persistence claim remains tied to this one setup.
  2. The comparison between tasks with and without reference answers changes several things at once, including the domains, system prompts, and option-construction procedures. The constructed PrinciplismQA action pairs also received no independent expert validation. Since the protocol forces a choice even between closely balanced alternatives, the validity of those pairs matters before a later change is read as sycophancy. The Easy/Hard distractor analysis neither controls these differences nor establishes comparable initial uncertainty.
  3. Final Flip conflates susceptibility under pressure with retention afterwards. For Gemini with Medium reasoning, conditional retention is 6.25% on both SV-ECHR and LegalBench (3/48 and 1/16, respectively), despite different Final Flip rates. The components measure different things, so the medical and legal rankings could reflect either.
  4. The experimental grid is not a full factorial: the reasoning-setting comparison is run under Bare pressure only, so reasoning-mode conclusions inherit the same construction choices as the pressure comparisons. All persistence evidence also rests on one fixed six-exchange construction, and a second set of inserted exchanges or a different reevaluation delay would show whether the effect is a property of the protocol or of the particular inserted exchanges.
Questions for authors
  1. Do you have results where every item, including no-flip and no-pressure paths, receives the same inserted exchanges and final question? What happens if the pressure messages are removed but the model's own responses are retained?
  2. What evidence beyond author review supports the balance of the constructed PrinciplismQA pairs? The archive contains item identifiers but no transformed scenario or option files. Releasing the non-sensitive constructed items would let these judgments be checked.
  3. Could you report conditional retention rates with uncertainty, separate from Final Flip, so the two components can be distinguished?
  4. What happens if the inserted exchanges are varied rather than fixed at one set of six?
5 comments

(W1, Q1) Additional control baselines

We thank the reviewer for the careful assessment of our controls, metrics, and experimental coverage. Please find our responses below.

W1. I am not convinced the reevaluation setup separates persistence from continued conditioning on the conversation. The pressure statements and the model's revised explanation stay visible, the protocol uses one fixed set of six inserted exchanges, and paths without a flip never receive the final question, so Neutral is not a complete control for changes caused by the reevaluation procedure. The authors acknowledge the history limitation in Section 7, but without a matched no-pressure reevaluation or an ablation of the retained history, the persistence claim remains tied to this one setup.

Q1. Do you have results where every item, including no-flip and no-pressure paths, receives the same inserted exchanges and final question? What happens if the pressure messages are removed but the model's own responses are retained?

A1. We appreciate your suggestions for additional controls.

As suggested, we extended reevaluation to previously unchanged Bare rebuttal paths and all Neutral paths using the same six exchanges and final question. The use of intervening auxiliary tasks follows Castillo-Bolado et al. (2024). New changes were rare in both controls:

ModelBare: previously unchanged paths (%)Neutral: overall final change (%)
GPT-5.6 Sol0.38 (3/799)0.83 (10/1,199)
Claude Opus 50.27 (1/368)0.25 (3/1,197)
Qwen 3.7 Flash0.00 (0/88)1.09 (13/1,198)
DeepSeek V4 Flash 07311.15 (3/261)0.85 (10/1,173)
Gemini 3.7 Flash0.54 (6/1,105)1.34 (16/1,198)

Bare: new changes among previously unchanged paths. Neutral: final changes among all valid paths. Pooled across eight benchmarks; failed/refused paths excluded. Full results are provided in Appendix D.6.

As a supplementary analysis of the history’s contribution, we removed user-pressure messages while retaining the model’s rationales and verdicts. Qwen 3.7 Flash (High) had a flip retention rate of 43.1% [40.7, 45.7] with pressure retained and 5.9% [4.7, 7.2] with pressure removed. DeepSeek V4 Flash 0731 (High) had corresponding rates of 73.8% [71.4, 76.2] and 4.8% [3.6, 6.1].

Three-run means [95% bootstrap CI]. Included paths: Qwen 1,033/1,109; DeepSeek 668/913, requiring valid responses in all four history conditions across all three repetitions.

Thus, consistency with earlier model responses alone does not explain the observed retention; retained user pressure also contributes to persistence within the conversation.

(W2) Clarification of experimental settings

W2. The comparison between tasks with and without reference answers changes several things at once, including the domains, system prompts, and option-construction procedures. The constructed PrinciplismQA action pairs also received no independent expert validation. Since the protocol forces a choice even between closely balanced alternatives, the validity of those pairs matters before a later change is read as sycophancy. The Easy/Hard distractor analysis neither controls these differences nor establishes comparable initial uncertainty.

A2. We appreciate your comments on comparability. We clarify that we kept the choice format, general judgment instructions, pressure prompts, and evaluation procedure common across both task groups.

Domains: Domain and subjectivity are closely related, making some differences unavoidable. We selected domains to broaden coverage beyond any single field.

System prompts: Domain-specific judgment criteria differ. The shared instructions require judgments based on the supplied facts and criteria, reconsideration only when new information or a specific error changes their application, and a rationale followed by an A/B verdict.

Options: Construction procedures depend on the source datasets. To parallel correct/incorrect alternatives, tasks without reference answers use opposing judgments or mutually exclusive actions wherever possible. Generated options require neutral wording and comparable specificity.

Easy/Hard: When converting tasks with reference answers into binary choices, we examined how distractor plausibility affected responses to user pressure rather than selecting an incorrect option at random. This analysis was not intended to control differences between task groups or match their initial uncertainty. Details are provided in Appendix E.2.


(W2, Q2) Validation of PrinciplismQA pairs

Q2. What evidence beyond author review supports the balance of the constructed PrinciplismQA pairs? The archive contains item identifiers but no transformed scenario or option files. Releasing the non-sensitive constructed items would let these judgments be checked.

A3. We appreciate your suggestions on validation and data access. We updated the supplementary material to include all 150 evaluated PrinciplismQA inputs: scenarios, questions, A/B choices, and source IDs. We apologize for their omission from the initial submission. During construction, the prompts required excluding obviously dominated alternatives, using neutral wording, and matching specificity across options. The full screening, generation, and review prompts are provided in Appendix B.3.

Each item has now received three evaluations. Two evaluators from our laboratory, who are not authors of this paper, each assessed all 150 items. Two initial evaluators provided the third evaluation by reviewing 75 items each.

Evaluators read the scenario, question, and both options, answered Yes or No for the following criteria, and could leave written comments:

(1) Question relevance: Do both options answer the question?
(2) Scenario grounding: Are both options grounded in the scenario?
(3) Mutual exclusivity: Do the options describe actions that cannot both be taken?

The pairs have not been validated by clinicians or ethicists. These checks required no medical knowledge or clinical judgment. We examined only the construction of the options.

CriterionMajority Yes (%)All Yes (%)
Question relevance100.0 (150/150)90.0 (135/150)
Scenario grounding100.0 (150/150)89.3 (134/150)
Mutual exclusivity98.0 (147/150)85.3 (128/150)

Majority Yes: at least two of three evaluations were Yes. All Yes: all three were Yes.

A pair passed if at least two of its three evaluations rated all three criteria Yes. Under this rule, 147 pairs (98.0%) passed; the remaining three were flagged for mutual exclusivity. Readers can inspect the complete set of options in the supplementary material. We will publicly release the code and evaluation data upon acceptance.

(W3, Q3) Flip retention rate

W3. Final Flip conflates susceptibility under pressure with retention afterwards. For Gemini with Medium reasoning, conditional retention is 6.25% on both SV-ECHR and LegalBench (3/48 and 1/16, respectively), despite different Final Flip rates. The components measure different things, so the medical and legal rankings could reflect either.

Q3. Could you report conditional retention rates with uncertainty, separate from Final Flip, so the two components can be distinguished?

A4. We appreciate your suggestion to report retention separately. We agree that Final Flip (F/NvalidF/N_{\mathrm{valid}}) reflects both and does not measure either independently. Any Flip (A/NvalidA/N_{\mathrm{valid}}) measures sycophantic switching under unsupported pressure, while the flip retention rate (Rflip=F/AR_{\mathrm{flip}}=F/A) — we additionally define — measures persistence after a flip.

Gemini 3.7 Flash (Medium)AnyFlip (%)FinalFlip (%)RflipR_{\mathrm{flip}} (%) [95% CI]
SV-ECHR32.002.006.25 (3/48) [2.15, 16.84]
LegalBench10.670.676.25 (1/16) [1.11, 28.33]

Bare rebuttal; Nvalid=150N_{\mathrm{valid}}=150 valid paths per dataset. AA: paths flipped under pressure; FF: paths retaining the change at reevaluation. Wilson 95% CIs.

For the other four models, the flip retention rates are numerically higher on tasks without reference answers in both medical and legal comparisons. Table 3 of the revised manuscript reports these results.

In our revised version, Section 3.5 defines the metrics; Appendix D.5 and supplementary/d_main_outcomes.csv provide the full results and confidence intervals. Incomplete paths are excluded; estimates and CIs are N/A when A=0A=0.


(W4) Ablations of reasoning-setting under pressures

W4. The experimental grid is not a full factorial: the reasoning-setting comparison is run under Bare pressure only, so reasoning-mode conclusions inherit the same construction choices as the pressure comparisons. All persistence evidence also rests on one fixed six-exchange construction, and a second set of inserted exchanges or a different reevaluation delay would show whether the effect is a property of the protocol or of the particular inserted exchanges.

A5. We appreciate your question about reasoning-setting coverage. We clarify that the reasoning-setting comparison is reported in Appendix D.3 in the submission version. Qwen’s High reasoning versus reasoning-disabled comparison covers all seven conditions and eight benchmarks, as presented in Appendix D.3, Figure 5. This comparison examines reasoning-setting effects across pressure conditions beyond Bare rebuttal.


(W4, Q4) Ablations of the inserted exchanges

Q4. What happens if the inserted exchanges are varied rather than fixed at one set of six?

A6. We appreciate your suggestion to vary the inserted exchanges. Persistence remained across exchange settings: flip retention rates varied, but the model ranking was unchanged. We use Distraction to assess whether changed judgments persist after intervening unrelated conversation. This ablation tests whether that persistence depends on the particular six exchanges selected.

ModelFlipped pathsNone (0)Fixed (6)Sampled (6)Combined (12)
GPT-5.6 Sol (Medium)40087.5084.5086.7582.00
Qwen 3.7 Flash (High)1,11039.7342.9747.9343.78
DeepSeek V4 Flash 0731 (High)91375.9074.7075.1475.58
Gemini 3.7 Flash (Medium)9337.6312.9027.9633.33

RflipR_{\mathrm{flip}} (%); parentheses: exchange pairs. The pressure history is held fixed. Fixed reuses Main; Sampled uses item-specific TriviaQA pairs shared across models; Combined appends them to Fixed. One Final Judgment per item per alternative. Appendix F.2 reports final-change rates over all matched paths.

These results show that persistence is not confined to the original six-exchange configuration.

Reviewer_cZb2OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SOLIPS (sycophancy on LLMs' initial judgements in pressure sequences), a protocol used for measuring changes/ sycophancy in decisions that: (i) have no single right answer (moral dilemmas, medical ethics conflicts, and split legal cases); (ii) have a single right answer. The motivation for this paper was that since existing sycophancy benchmarks judge a model’s answer against a ground truth, they cannot tell whether a change of mind on such tasks is sycophantic or legitimate. As such, SOLIPS measures whether a model abandons its own initial choice for the option the user demands, when the user gives no actual reason to change. It also tests whether that change persists after the pressure stops, which prior work has largely not examined.

In testing, the tested model receives each problem defined as a binary choice along with explicit judgment criteria. After the model’s initial answer, the user pushes back for up to three rounds under one of seven conditions (neutral, emotion-backed, evidence-backed, etc.), until the model switches its answer. Then, six unrelated trivia exchanges are inserted and the original question is asked again with the conversation history kept. The authors evaluate five models under two reasoning settings on eight datasets of 150 problems each. Of the datasets, four have no reference answer, and four do.

Results show that five models are more likely to keep their switched answer on medical and legal tasks without a reference answer than on comparable tasks with one. Models that switch equally often can also differ widely in persistence. Additionally, some switches only appear to occur in the second or third round, and adding persuasive wording to the rebuttal does not increase switching beyond the plain rebuttal. Those results show that sycophancy evaluations for decision support should include tasks without reference answers, measure whether changes persist after pressure is removed, and report results by pressure condition and round.

Critical strengths
  1. The paper addresses an important gap in sycophancy evaluations by incorporating a reevaluation step and questions with no correct answer. In other words, existing benchmarks need a correct answer to decide whether a change of mind is sycophantic, and they only evaluate a 'flip' at a singular stage, while this paper adds those 2 extra layers.

  2. Well controlled and clean experimental design, including standardized pressure rebuttals with minimal changes relative to the plain rebuttal, thus isolating the effect of extra wording. This allowed for the conclusion that persuasive framing adds nothing beyond simply telling the model it is wrong.

Critical weaknesses
  1. Despite not having a "correct answer choice" in four datasets being a core contribution of this paper, it adds a layer of weakness in which we cannot distinguish sycophancy from legitimate uncertainty, which undermines its main comparison. For instance, on a difficult philosophical dilemma, the model’s first choice may be close to arbitrary, so switching under pressure may simply reflect that both options are defensible. In other words, the finding that models switch and stay switched more often on these tasks may therefore mean only that these are closer calls, not that models are more sycophantic on them. One way to measure how stable or confident the initial model choice was would be, for example, by resampling the initial answer several times.

  2. The Final Flip measure doesn’t clearly show whether a change of mind lasts, which is the paper’s second main contribution. When the original question is asked again, the model can still see the whole earlier conversation, including the user’s pushback and its own switched answer. If it gives the switched answer again, it may simply be staying consistent with what it just said, not showing a lasting change in judgment. Final Flip is also calculated over all problems rather than only the ones where the model switched, so it mixes how often a model switches with how often it keeps the switch. Re-asking the question in a fresh conversation, and reporting the share of switches that are kept, could make this measure much easier to interpret, although that may have other limitations.

Questions for authors
  1. Building up on my second point in the critical weaknesses, what happens to Final Flip if the original question is re-asked in a fresh conversation, without the earlier pushback and switched answer visible? Could you also report the share of switches that are kept (Final Flip divided by Any Flip) for each model and dataset? This would show whether the changes actually persist or whether the model is just staying consistent with its last message.

  2. Were the LLM-generated PrinciplismQA action pairs checked for balance, for example by having clinicians or ethicists rate whether both options are defensible and neither is clearly better?

4 comments

(W1) Initial judgement stability

We thank the reviewer for the constructive comments on initial judgment stability, persistence, and action-pair validation. Please find our responses below.

W1. Despite not having a "correct answer choice" in four datasets being a core contribution of this paper, it adds a layer of weakness in which we cannot distinguish sycophancy from legitimate uncertainty, which undermines its main comparison. For instance, on a difficult philosophical dilemma, the model’s first choice may be close to arbitrary, so switching under pressure may simply reflect that both options are defensible. In other words, the finding that models switch and stay switched more often on these tasks may therefore mean only that these are closer calls, not that models are more sycophantic on them. One way to measure how stable or confident the initial model choice was would be, for example, by resampling the initial answer several times.

A1. We appreciate your suggestion to assess initial judgment stability. We collected four additional Initial Judgments per item, yielding five responses including the Main initial response. Models still switched under unsupported pressure on items where all five initial choices agreed, so initial inconsistency alone cannot explain the observed switching.

ModelReference answer5/5 (%)Any Flip, all (%)Any Flip, 5/5 (%)
GPT-5.6 Sol (Medium)w/98.7 (591/599)12.9 (77/599)11.8 (70/591)
Claude Opus 5 (High)w/98.7 (590/598)69.6 (416/598)69.2 (408/590)
GPT-5.6 Sol (Medium)w/o86.5 (519/600)53.8 (323/600)47.8 (248/519)
Claude Opus 5 (High)w/o92.7 (556/600)68.9 (413/599)66.7 (370/555)

5/5: identical choices across all five Initial Judgments. Any Flip: Bare rebuttal; all valid items or the 5/5 subset.
Four datasets per task group. One consistent Claude item was excluded from Any Flip due to provider refusal.

Initial consistency was lower on tasks without reference answers. Nevertheless, GPT’s higher switching rate on these tasks persisted within the consistent subset, so this pattern cannot be explained solely by a larger proportion of initially inconsistent items. Claude switched frequently in both groups, showing that consistent initial choices do not guarantee resistance to pressure.

Dataset-level Fleiss’ κ ranged from 0.808–1.000 for GPT and 0.898–1.000 for Claude. In our revised version, results for all five models are provided in Appendix F.4. These findings support SOLIPS’s evaluation of whether models maintain their judgments under unsupported user pressure, beyond how consistently they answer the initial question.

(W2, Q1) Flip retention rate

W2. The Final Flip measure doesn’t clearly show whether a change of mind lasts, which is the paper’s second main contribution. When the original question is asked again, the model can still see the whole earlier conversation, including the user’s pushback and its own switched answer. If it gives the switched answer again, it may simply be staying consistent with what it just said, not showing a lasting change in judgment. Final Flip is also calculated over all problems rather than only the ones where the model switched, so it mixes how often a model switches with how often it keeps the switch. Re-asking the question in a fresh conversation, and reporting the share of switches that are kept, could make this measure much easier to interpret, although that may have other limitations.

Q1. Building up on my second point in the critical weaknesses, what happens to Final Flip if the original question is re-asked in a fresh conversation, without the earlier pushback and switched answer visible? Could you also report the share of switches that are kept (Final Flip divided by Any Flip) for each model and dataset? This would show whether the changes actually persist or whether the model is just staying consistent with its last message.

A2. We appreciate your questions about persistence and fresh conversations. In our setup, prompting does not update model parameters, and fresh conversations carry no prior state. Thus, judgment changes do not persist beyond the current context. SycEval studies persistence across rebuttals. Here, persistence means retaining the changed choice at Final Judgment within the same conversation. Consistency with earlier outputs can contribute to this persistence. However, retaining model responses while removing user pressure reduced Qwen 3.7 Flash’s flip retention rate from 43.1% to 5.9% (three-run means over 1,033 complete paths). Thus, consistency with prior outputs alone does not explain persistence. A fresh conversation uses the same input as Initial Judgment, so we address this question using the repeated Initial Judgments reported in A1.

We report the requested share as the flip retention rate, Rflip=F/AR_{\mathrm{flip}} = F/A counts paths changed under pressure, and FF those retaining the change at Final Judgment. Representative results by model and dataset are shown below.

ModelPrinciplismQAMedQA-USMLESV-ECHRLegalBench
GPT-5.6 Sol (Medium)84.88 (73/86)68.75 (11/16)84.78 (117/138)79.55 (35/44)
Claude Opus 5 (High)91.60 (109/119)28.46 (37/130)96.00 (144/150)63.95 (94/147)
Qwen 3.7 Flash (High)72.67 (109/150)27.94 (38/136)54.00 (81/150)37.12 (49/132)
DeepSeek V4 Flash 0731 (High)84.56 (126/149)46.39 (45/97)91.89 (136/148)72.59 (98/135)
Gemini 3.7 Flash (Medium)31.25 (5/16)20.00 (1/5)6.25 (3/48)6.25 (1/16)

Bare rebuttal. Rflip=F/AR_{\mathrm{flip}} = F/A, % (F/A). Incomplete paths are excluded; estimates are N/A when A = 0.

Changed choices persisted more often on tasks without reference answers in nine of ten comparisons. Thus, evaluation should distinguish switching under pressure from subsequent persistence. The retention rate shows how often a changed choice persists at reevaluation.

We added the definition and representative results to Section 3.5 and Table 3. Appendix D.5 provides results for all five models and eight datasets under Bare rebuttal at both reasoning settings, with Wilson 95% confidence intervals.

(Q2) Validation of PrinciplismQA pairs

Q2. Were the LLM-generated PrinciplismQA action pairs checked for balance, for example by having clinicians or ethicists rate whether both options are defensible and neither is clearly better?

A3. We appreciate your suggestion to validate the constructed action pairs.During construction, the prompts required excluding obviously dominated alternatives, using neutral wording, and matching specificity across options. The full screening, generation, and review prompts are provided in Appendix B.3.

We undergo a human review of all 150 pairs, with three evaluations per item. Evaluators read the scenario, question, and both options, answered Yes or No for the following criteria, and could additionally leave written comments:

(1) Question relevance: Do both options answer the question?
(2) Scenario grounding: Are both options grounded in the scenario?
(3) Mutual exclusivity: Do the options describe actions that cannot both be taken?

Since these questions required no medical knowledge or clinical judgment, we recruited four university graduates.

CriterionMajority Yes (%)All Yes (%)
Question relevance100.0 (150/150)90.0 (135/150)
Scenario grounding100.0 (150/150)89.3 (134/150)
Mutual exclusivity98.0 (147/150)85.3 (128/150)

Majority Yes: at least two of three evaluations were Yes. All Yes: all three were Yes.

A pair passed if at least two of its three evaluations rated all three criteria Yes. Under this rule, 147 pairs (98.0%) passed; the remaining three were flagged for mutual exclusivity. We will update evaluation results without the invalid evaluation samples in camera-ready.

We updated the supplementary material to include all 150 evaluated inputs: scenarios, questions, A/B choices, and source IDs. We will publicly release the code and evaluation data upon acceptance.

As the author–reviewer discussion period is coming to a close, we would appreciate hearing any remaining concerns you may have. We would be happy to provide further clarification. Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We appreciate the valuable feedback from all reviewers. Reviewers cZb2 and Rikm recognized the importance of evaluating judgments without a single correct answer, while Reviewer ghUf highlighted our broad empirical coverage. Our study enables comparisons across tasks with and without reference answers while distinguishing susceptibility to pressure from subsequent persistence.

To address concerns about initial stability and reevaluation (cZb2, W1; Rikm, W1/Q1), we repeated Initial Judgment and added reevaluation controls. Switching remained substantial among consistently answered items, while changes were rare in the controls. These findings strengthen the evidence for susceptibility to unsupported pressure and the stability of our reevaluation procedure.

To assess the constructed PrinciplismQA options (cZb2, Q2; Rikm, W2/Q2), we completed three evaluations per item. Under the stated majority rule, 147 of 150 pairs met all three criteria: question relevance, scenario grounding, and mutual exclusivity. This reduces concern that the observed judgment changes arose from basic flaws in option construction.

We separately analyzed retention, varied Distraction, and examined returned rationales and flip timing (cZb2, W2/Q1; Rikm, W3/Q3 and W4/Q4; ghUf, W2/Q1). Together, these analyses show that susceptibility and persistence capture different behaviors and help interpret how changed judgments are maintained or reversed.

We address the detailed comments in our individual responses and will incorporate the revisions and additional results into the camera-ready version if accepted.

Thank you once again for your valuable feedback on our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.