acceptodds
Under review as a conference paper at ICLR 2027

Why LLMs Are Sycophantic

37 views

Abstract

Large language models (LLMs) often abandon a correct answer, or endorse a user's position, once the user pushes back. This behavior, called sycophancy, is usually reported as a single rate per model, which says little about when it happens or how a user can avoid it. We study the conditions that produce it with 103,939 graded replies from ten configurations: eight LLMs with reasoning disabled, and two of them again with maximum reasoning, all facing the same 200 items, 13 pressure conditions, and four-turn conversations, with every reply labeled by two independent LLM judges. We find that the dominant factors are how costly it is for the model to verify the user's claim, and whether a trained guardrail covers it. Removing this task factor from a logistic model costs 0.485 of McFadden , against 0.139 for model family and 0.009 for pressure tactic. Anchored facts are almost never conceded (1.3%), while adoption on logic puzzles rises with the number of clues needed to refute the pushed answer. Personal choices are endorsed in 77.0% of conversations. Most concessions on hard items come from models that cannot reliably solve them; models that can solve them rarely give the answer up. For both models tested, maximum reasoning removes these concessions completely: adoption on deep puzzles falls from 19.2% and 12.5% to 0%. Fallacious or emotional framing adds nothing beyond plain repetition. Three human annotators agree with the judges' consensus on 118/120 calibration items. These results give practical rules for reliable use: simplify hard-to-verify problems and reason deeply, state the question rather than one's preferred answer, ask for evidence on open questions, and choose models by their measured guardrail profile.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM sycophancy, language model alignment, verification cost, model behavior, adversarial dialogue, epistemic deference, guardrails, large language model evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_v6UbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper investigates when language models adopt a user's asserted answer or position under repeated conversational pressure. It evaluates eight models with reasoning disabled and two of them again at maximum reasoning, using 200 items, 13 pressure conditions, and four-turn conversations. Its main methodological contribution is a solver-certified, claim-specific measure of refutation size, min-k, evaluated through matched shallow/deep puzzle questions and complementary linear-system tasks. It also conditions adoption on independently estimated item-level capability. The authors report strong task-region differences, greater adoption on difficult-to-refute claims, higher adoption in low-capability strata, and no observed reasoning-item adoptions in the two maximum-reasoning configurations. Two LLM judges and a blinded human calibration support the reliability of the commitment labels.

Critical strengths

1.The design—using a solver to determine how many clues are needed to refute an incorrect answer—is quite interesting.

2.The evaluation is careful, with two judges, human annotation, and useful robustness checks

Critical weaknesses

1.Both options in the personal-choice tasks are reasonable. Supporting the option the user prefers is not necessarily sycophancy. This makes the high adoption rate in this category hard to interpret.

2.The shallow and deep questions share the clues, but ask about different answers. Puzzle size also changes across min-k levels. It is still unclear how much of the effect comes from verification cost rather than general task difficulty.

3.The capability analysis excludes the intermediate group, which accounts for about 60% of all deep-puzzle adoptions. The results show a difference between the extreme groups, but do not establish that most concessions are deference.

4.Maximum reasoning is tested on only two models, I think this is limited.

5.I do not believe this paper proposes a truly novel architecture to explain sycophancy in LLMs. Instead, it focuses on rather obvious factors—such as pressure, the specific model used, and the difficulty of verification—that clearly influence sycophantic behavior. In my view, the paper lacks innovation.

Questions for authors
  1. Does the region/model/pressure ordering remain, with comparable effect sizes, when reasonable personal-choice endorsement is excluded or analyzed separately? What evidence distinguishes a model's appropriate accommodation of a preference from the sycophancy interpretation used in the paper?

2.Please distinguish what the shallow/deep matching identifies from what remains confounded. If these effects cannot be separated, would you narrow the contribution to a certified structural association rather than an isolated verification mechanism?

3.The missingness is concentrated on difficult items. How should the zero-adoption result be interpreted over all attempted tasks?

  1. If the author insist I have some misunderstanding of your innovation, please clarify your innovation refer to my weakness.
6 comments
AuthorsOpenReview ↗

Response to Reviewer v6Ub (1/4): W1

Thank you for your careful review and your detailed questions. We are also glad that you found the solver-based design interesting and the evaluation careful. We answer each weakness (W1 to W5) and then each question (Q1 to Q4), in four comments. All analyses use the runs reported in the paper.

W1: We agree that endorsing a reasonable option the user prefers is not an error in itself. We will report this region as preference endorsement, separate from error adoption.

We included this region deliberately, as one end of a spectrum. At one end are questions with a checkable answer, where agreeing with a wrong claim is clearly an error. In the middle are questions with normative weight, such as contested policy and moral scenarios. At the other end are personal choices, where no option is wrong and nothing in the question resists agreement. This end shows how strong the tendency to agree with the user is when nothing holds it back. Comparing it with the other regions shows how much a checkable answer or a norm reduces that tendency. It also reflects a common use. People often ask models for advice on everyday decisions where no shared standard exists and judgments vary from person to person. This is where agreement is hardest to check and most likely to go unnoticed. The two analyses requested in Q1 address the interpretation directly (see Q1).

Revision: We will report personal choices as preference endorsement and add both analyses from Q1. In the Discussion, we will explain why we include this region: it is one end of the spectrum, and it matters for everyday decisions where no shared standard exists.

AuthorsOpenReview ↗

Response to Reviewer v6Ub (2/4): W2

W2: We agree. Within a pair, the deep question is also harder to solve, and min-k rises with puzzle size (r = 0.99). The matched pairs hold the puzzle, clues, topic, and answer format fixed, but they cannot separate checking cost from solving difficulty.

Separating the two requires tasks where they move apart. Linear systems do this. With more unknowns, a system becomes longer and more error-prone to solve by hand, yet a pushed answer is still refuted by one substitution. We measured both costs as the tokens a model spends to solve an item (turn 1) and to check the pushed answer (turn 2). Token counts require a maximum-reasoning run, and we use GPT-5.6-Luna because Claude-Sonnet-5's run is incomplete on the hardest items.

Table 1. GPT-5.6-Luna: median tokens at maximum reasoning, and adoption with reasoning off.

ItemsSolve (turn 1)Check (turn 2)Adoption
Common facts1382180.5% (3/648)
Shallow puzzles (min-k 1)1253440.0% (0/156)
Deep puzzles (min-k 4)3344903.6% (1/28)
Linear systems (4 or 6 unknowns)1,8346321.7% (4/233)
Linear systems (8 unknowns)8,2737190.0% (0/104)
Deep puzzles (min-k 8)5,8202,78118.8% (6/32)
Deep puzzles (min-k 12 or 16)32,82610,33513.5% (7/52)

The key comparison is between eight-unknown systems and min-k 8 puzzles. The systems take more tokens to solve but far fewer to check, and they draw no adoption, while the puzzles draw 18.8%. Within linear systems, solving cost rises more than fourfold while checking cost barely changes, and adoption does not rise. Returning to deep puzzles, this model's adoption stops rising above min-k 8. With reasoning off, it solves only 24.4% of them, so beyond that depth it can neither solve nor check. The counts are also small (32 and 52 conversations). Pooled over all eight models, adoption rises from 6.4% at min-k 1 to 48.0% at min-k 12 or 16 (Section 4.3).

For this model, each doubling of checking tokens multiplies the odds of adoption by 2.1 (95% CI 1.8 to 2.5), while solving tokens add nothing (odds ratio 0.76, p = 0.2). The effect of checking tokens remains after controlling for the model's solve rate (odds ratio 2.1, CI 1.6 to 2.7). The same pattern holds beyond this model. Pooled over all eight models, using each model's own reply length in place of tokens gives the same result (odds ratio 2.03 per SD, solve rate controlled, CI 1.43 to 2.88).

Holding inability to solve fixed gives the same picture. Among items a model cannot solve (correct in at most 20% of its other conversations), adoption is 35.5% (114/321) on deep puzzles but 9.9% (141/1,420) on linear systems. Within models, unsolvable deep puzzles have 6.4 times the odds of adoption of unsolvable linear systems (95% CI 4.0 to 10.2).

Because puzzles and linear systems also differ in format, we treat these comparisons as supporting evidence rather than proof. A cleaner test stays within one task type. Across linear systems with 4, 6, and 8 unknowns, solving difficulty rises while checking stays at one substitution. Unlike for GPT-5.6-Luna alone, adoption pooled over all eight models rises from 2.1% at 4 unknowns to 14.4% at 8. Once we control for the model's solve rate on the item, system size no longer predicts adoption. So the ability to solve also matters, even when the check is cheap, and both factors contribute. The revised paper will report adoption and turn-1 solve rates by system size for each model.

Revision: We will remove "isolates," describe the min-k result as a certified structural association (see Q2), add this analysis to Section 4.3, and state the remaining confounds explicitly. We will keep min-k as our measure of verification cost, because it is certified by a solver and defined before any model is run. We will report checking tokens as a complementary, model-specific measure that agrees with it.

AuthorsOpenReview ↗

Response to Reviewer v6Ub (3/4): W3, W4, W5

W3: We agree. A difference in rates between the two extreme groups does not show that most concessions are deference. We counted all final-turn concessions on deep puzzles, with no group excluded. Groups follow Section 4.4. Solvable means the model solves the item in at least 80% of its other 12 conversations. Unsolvable means at most 20%, and intermediate is everything in between.

Table 2. Final-turn concessions on deep puzzles, eight models, reasoning off.

GroupAdoption rateShare of concessions
Solvable3.5% (11/310)3.6%
Intermediate44.8% (184/411)59.5%
Unsolvable35.5% (114/321)36.9%

Unsolvable items account for 36.9% of concessions, not a majority, so we will no longer claim that most concessions are deference. What the data support is narrower: almost no concessions come from items the model solves reliably. Only 3.6% do; the other 96.4% come from items the model solves in fewer than 80% of its own conversations.

The intermediate group reflects unstable capability rather than knowledge the model holds. When a model in this group answered a deep puzzle correctly at turn 1, it still gave up that answer in 30.7% of conversations (46/150), close to the unsolvable group's rate.

Revision: We will report all three groups in Section 4.4, retitle it "Concessions come from items the model cannot reliably solve," and state the claim for deep puzzles only. We will make the same change in the abstract, the introduction, and Figure 1, replacing "hard items" with "deep puzzles."

W4: We agree that two models limit how far this result generalizes. Running at maximum reasoning is expensive. On the hardest puzzles, a single reply can need more than 30,000 tokens (median 32,826 for GPT-5.6-Luna at min-k 12 or 16). Some of Claude-Sonnet-5's replies exceeded our 64,000-token budget. Across all items and 13 conditions, each maximum-reasoning configuration therefore costs far more than a reasoning-off run, so we limited this comparison to two models from different developers.

The two models start far apart: with reasoning off, they solve 69.9% and 24.4% of deep puzzles. At maximum reasoning, both solve nearly all of the puzzles they complete. Neither concedes on any reasoning item where it completed its own turn-1 answer (0/548 and 0/648; see Q3). The result is consistent with Section 4.4: once a model can solve an item, it can check the pushed answer.

Revision: The revised paper states this result only for the two models tested and lists this limitation in the Discussion.

W5: Our contribution is a measurement framework for sycophancy. Each factor may seem intuitive on its own, but how much each one matters, relative to the others, had not been measured. We measure all four on the same items, turns, and judges, with a solver-certified measure of checking cost (min-k). Several results were not expected:

  1. Pressure wording adds little. On average, eight fallacies and four emotional framings yield no more adoption than plain repetition (12.9% and 13.2% vs. 15.0%). Only the appeal to popularity exceeds it, by 2 to 3 points.
  2. Checking cost matters beyond solving difficulty. Among items a model cannot solve, deep puzzles draw 35.5% adoption, while linear systems, which are cheap to check, draw 9.9% (W2).
  3. Concessions rarely come from items the model reliably solves (3.6% on deep puzzles, W3).
  4. A certified measure of checking cost. As you noted, min-k is computed and verified by a solver for each claim, which, to our knowledge, prior benchmarks do not provide.

These results are also practical: they suggest choosing models by their measured profile, splitting problems into steps that are cheap to check, and asking without a preferred answer.

Revision: We will state these contributions explicitly in the introduction, and release all items, transcripts, labels, and code.

Reviewer_AwErOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

the paper asks WHY LLMS ARE SYCOPHANTIC. The authors find that this depends mostly on the type of question (for example, simple facts versus hard puzzles versus personal choices), much less on which LLM is used, and almost not at all on how the user pushes.

Critical strengths
  • The paper asks an important question and tests it on a large, carefully built set of questions.
Critical weaknesses
  1. The title asks "why" the LLMs give in, but the paper only shows "when." The main comparison is between different types of questions, and these differ in many ways at once (topic, difficulty, kind of answer). So the paper cannot tell which of these differences actually causes the behavior. about the "when" for example also other papers like https://arxiv.org/pdf/2606.16011 also shown in table 5 when the domain is STEM then for exampel the syphocancy is less.
  2. The paper says the cause is the way LLMs are trained to please people. It was valid in 2023-2024 when we always did the RLHFs but current LLMs are beyond that.
  3. The paper explains some results by "trained safety rules" in the LLMs, but it never checks whether these rules exist. It simply assumes them based on the type of question, so the explanation goes in a circle.
Questions for authors
  • I think a different title would describe the paper more accurately.
  • The paper should add more work in this area. I know at least 10 more papers that are directly related, and some of them would help support the paper's claims.
4 comments
AuthorsOpenReview ↗

Response to Reviewer AwEr (1/2): W1, W2

We are grateful for your careful reading and your constructive comments. Your assessment that the paper asks an important question on a carefully built item set means a great deal to us. Our answers follow your numbering, and every number comes from the runs reported in the paper.

W1: We agree. The paper shows when models yield, not why. We will change the title accordingly, and we would welcome your suggestion for the wording. We will also state in the introduction that the paper measures conditions and does not identify causes.

We also agree that task types differ in topic, difficulty, and kind of answer at the same time, so a comparison across task types cannot identify a cause. Our data allow narrower comparisons that hold more factors fixed. Two are already in the paper, and we will bring all three forward and state what each one holds fixed.

Table 1. Comparisons that hold more factors fixed.

ComparisonHeld fixedWhat changesFinal-turn adoption
Common facts, civics vs. STEMtask type, kind of answer, one lookup to checktopic0.6% and 1.9%
Same puzzle, shallow vs. deep question (Section 4.3)topic, clue text, kind of answer, model, pressureclues needed to refute the pushed answerhigher for the deep question in all eight models, by 7.3 to 43.1 points
Same model and items, reasoning off vs. maximum (Section 4.5)items, user turns, modelhow far the model can reasondeep puzzles: 19.2% and 12.5% with reasoning off; 0/118 and 0/154 at maximum

The last row counts conversations in which the model completed its own turn-1 answer.

Topic and kind of answer alone therefore do not produce the pattern. These comparisons narrow the question without settling the cause, so "when" is the accurate word.

Thank you also for pointing us to Nikeghbal et al. (2026). Their Table 5 shows that nine of the ten most stable MMLU subjects are STEM subjects, and the authors suggest that verifiable answers may explain this. This agrees with our results: civics and STEM facts can both be checked with one lookup, and models hold both. We will cite this paper and discuss it in Section 4.1.

Revision: New title; a statement in the introduction that we measure conditions, not causes; Table 1 in Section 4.1; citation and discussion of Nikeghbal et al. (2026).

W2: We agree that our paper cannot attribute sycophancy to a training method. Current models are trained with several methods, including reinforcement learning on verifiable tasks and feedback from AI judges, alongside preference-based training. The exact recipes are mostly not public. We did not study how any model was trained.

What our data do show is behavioral: the eight models differ widely on the same items, for example from 0.4% to 51.4% endorsement on contested policy. We report these differences without attributing them to a cause.

Revision: We will present the preference-training account as background from earlier work, and remove the statements in Sections 4.7 and 6 that attribute the behavior to training.

AuthorsOpenReview ↗

Response to Reviewer AwEr (2/2): W3, Q1, Q2

W3: We agree. The paper assigns a guardrail to each task type by assumption and never checks that it exists. We have now run this check. It uses only turn 1, before the user pushes any answer, so it does not depend on the results it is meant to explain.

Table 2. Turn-1 replies before any pressure, eight models.

Task typeRule assumed in the paperReplies that follow the ruleAcross models
Unknowable quantitysay the value is not known96.1%at least 84.9% for every model
Contested policytake no side83.3%at least 80% for seven of the eight models
Personal choiceno rule assumed44.7% take no side15.8% to 66.9%

Both rules are visible before any pressure. Models say that an unknowable value is not known, and seven of the eight models take no side on contested policy in at least 80% of replies. The check also refines one label. On the five moral-scenario items, models take a side at turn 1, so we will relabel them. Moral scenarios form their own region in the analysis, so the reported results do not change.

The check has a limit. It shows that these rules exist as default behavior, but not where they come from, so we will no longer call them "trained."

Revision: We will define each guardrail by the turn-1 behavior in Table 2, add the table to the paper, and relabel the moral-scenario items.

Q1: We agree, and we will change the title. Please see our answer to W1.

Q2: Thank you for this suggestion. We agree and regret the omission. We will cite and discuss the paper you pointed to (see W1). We would be grateful if you could share the other papers you have in mind. We will read them, cite them in Section 2, and state how our results relate to them.

The title change, the turn-1 check of the guardrails, and the broader related work all come from your comments, and the paper is more accurate for them. Thank you for the direct feedback and for your encouragement.

Thank you for all the responses. I have no suggestions regarding the title and acknowledge your answers to my other concerns. My positive view of the paper remains unchanged, and I continue to support its acceptance given the promised revisions.

Thank you for your continued support and for acknowledging our responses. We will incorporate all the promised revisions into the revised version of the paper.

Reviewer_s9u6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores the factors underlying sycophancy in language models. The authors models sycophancy as a function of the verifiability of a nudged claim, training guardrails relevant in context, the model being evaluated and a tactic used to apply pressure towards adoption of the claim. The authors hypothesize that these factors form a cascade: verifiability and guardrails describe the extent of yielding, while model and pressure tactics further nudge the rate relative to the task region described by the primary factors.

The authors examine the importance of these factors under a common experiment design, covering different tasks with varying levels of verifiability and applicable guardrails, models and pressure tactics, evaluated across a 44-turn conversation. They find that the task region, described by the verifiability and applicable guardrails is the most important factor for sycophancy, followed by the model. Pressure tactic has comparatively low importance, suggesting that tactics add nothing beyond repetition. Verifiability majorly informs adoption, with adoption rates rising with the depth of verification. Scaling test-time compute via reasoning improves verification, addressing the source of the majority of adoption cases. Taken together, the paper suggests that sycophancy is a property of the task and the model, with framing adding little effect.

Critical strengths
  • The paper studies the contribution of both task-level and model-level factors towards sycophancy under a single experiment design.
  • The analysis covers a diverse range of models and task settings.
  • Some of the findings are quite interesting, especially the relation of sycophancy with a claim's verifiability instead of the problem's relative difficulty.
Critical weaknesses
  • The role of pressure tactic may be understated. The authors consider final-turn adoption as the primary metric. However, most final-turn adoptions follow from the first push, where the variation across tactics is unclear as the figure is pooled across tactics. Successive repeated pushes and withdrawals may dilute the differences by the final turn, contributing to the low factor importance. The authors should mention that their finding applies to outcomes aggregated across turns, or report the tactic effects across other turns.
  • The effects of user-level recommendations are only partly tested, and some may not apply broadly. In practice, it may be difficult for users to guess whether a claim is hard to verify. Users may also have assumptions they might be unsure about, and would like a model to verify. They may also not know of any alternatives to compare against, and presenting multiple candidates which may all be wrong might anchor models towards an incorrect solution, even with reasoning. In such cases, the recommendations may be less practical than the paper suggests.
Questions for authors

Does the importance of factors change across turns?

4 comments
AuthorsOpenReview ↗

Response to Reviewer s9u6 (1/2): W1

We appreciate your thoughtful and constructive review. It is encouraging that you found the single experimental design, the range of models and tasks, and the verifiability results of interest. We respond to your two concerns and your question in turn, using only the runs reported in the paper.

W1: We agree. Our main estimate uses the final turn, while most adoptions begin at the first push. The paper already reports one related check. Counting adoption at any of turns 2 to 4 leaves the ordering unchanged, with tactic importance rising from 0.009 to 0.015 (Section 4.1). Following your suggestion, we have now also analyzed each turn separately.

Table 1. Each turn analyzed separately, eight models, reasoning off. The first three columns give the drop in McFadden R² when a factor is removed; the last three give adoption.

TurnTask regionModel familyPressure tacticPlain repetitionFallaciesEmotional framings
2 (first push)0.5020.1050.01917.1%13.4%15.0%
30.4880.1210.01216.2%13.1%13.9%
4 (paper)0.4850.1390.00915.0%12.9%13.2%

Your expectation is confirmed. The tactic matters about twice as much at the first push as at the final turn, and the differences then narrow. The spread across the 13 conditions falls from 10.7 points at turn 2 to 8.3 points at turn 4. One in five turn-2 adoptions is withdrawn by turn 4.

Even at the first push, however, the tactic remains the smallest factor by a wide margin, and fallacies and emotional framings on average stay below plain repetition. Only one of the twelve tactics, the appeal to popularity, is reliably above it, by 2 to 3 points.

Revision: We will add Table 1 to the paper, show the per-turn trajectories separately by tactic, and state that the main estimate refers to the final turn. We will also replace "adds nothing" with "adds little" in the abstract and in the title of Section 4.6.

AuthorsOpenReview ↗

Response to Reviewer s9u6 (2/2): W2, Q1

W2: We agree. These situations are more varied than our recommendations suggested, and no single practice fits every case. In the revision, we will replace the general list with practices matched to the type of question. We will present them as implications rather than rules, and mark which ones rest on a measurement.

Questions with a checkable answer (facts, puzzles, calculations):

  • Ask before stating an answer. Measured: every conversation asks the question alone at turn 1 and adds the user's answer from turn 2. On items with a correct answer, replies give the user's wrong answer in 1.8% of cases at turn 1 and in 8.4% at turn 2.
  • Use more reasoning. Measured on two models (Section 4.5).
  • Split hard problems into steps that are cheap to check. An implication of the verification-cost results (Section 4.3), not tested directly. It agrees with well-established methods for hard problems, such as least-to-most prompting (Zhou et al., ICLR 2023), decomposed prompting (Khot et al., ICLR 2023), and chain-of-verification (Dhuliawala et al., Findings of ACL 2024). These methods were designed to help models handle harder problems and reduce errors such as hallucinations. Our results add a new reason they may help. Cheaper checks go with fewer concessions, so splitting a problem into smaller sub-problems that are easier to check may also reduce sycophancy.

Opinion questions (contested policy, moral scenarios, personal choices):

  • Ask for the main perspectives and their evidence rather than a verdict. An implication: when asked for a verdict, models often endorse the user's stance (Sections 4.1 and 4.7). Here the options are known and none is wrong, so listing them leaves the judgment with the user.

For both:

  • Choose the model. Measured: the eight models differ widely on the same items (Section 4.7).

Your three cases show why this distinction matters:

  • An assumption the user wants checked. Our conversations test this case: the user states a claim and asks whether it is right. When the claim is cheap to check, models rarely adopt a wrong one (1.3% on facts, 6.4% on shallow puzzles). The risk lies where the check is costly (29.7% on deep puzzles).
  • Not knowing what is hard to verify. We agree that users often cannot tell. For this reason, we will lead with practices that do not require this judgment: asking first and using more reasoning.
  • Comparing candidates. We did not test this, and your point about anchoring is well taken. For questions with a correct answer, candidates that are all wrong could anchor the model, so we will drop this suggestion there. We will keep it only for opinion questions, where no option is wrong. There, the model should lay out the options and their evidence, so that the user chooses based on their own preferences.

Revision: We will rewrite the implications in Section 6 as practices matched to the type of question, and mark which are measured and which are implications. We will drop the comparing-candidates suggestion for questions with a correct answer.

Q1: The ordering of the three factors is the same at every turn: task region first, model family second, pressure tactic last. This also holds on the conversations labeled at every turn. The size of the tactic effect does change. It is about twice as large at the first push (0.019) as at the final turn (0.009), but it remains the smallest factor throughout (see W1 and Table 1).

Your suggestions made the paper clearer about what the pressure result covers and how far the recommendations go. We are happy to provide more detail on any point, and we thank you again for your support.

Thank you for your responses. I appreciate the additional results in Table 1. With the proposed revisions, all my concerns have been addressed, and I do not have any further questions. My view of the paper remains positive, and I support its acceptance.

Thank you very much for your support and for your recognition of our work. We are sincerely grateful. Thank you again for your valuable comments and suggestions. They made the paper clearer and stronger, and we will reflect them carefully in the revised version.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thank You to the SAC, AC, and All Reviewers

We sincerely thank the Senior Area Chair, the Area Chair, and all the reviewers for your time and for your careful reading of our paper. We are grateful for your support, and just as grateful for the problems you pointed out. We learned a great deal from your reviews. We will take all of your comments into account and use them to improve the paper.

The goal of this work is to understand sycophancy in LLMs. We will release all of our data and code. We hope they offer something useful to the community and help others understand this behavior better.

We also meet sycophancy in our own use of AI agents, and this is why we believe a careful analysis of it is important. This work measures four factors together: the type of question, the cost of checking a claim, the choice of model, and the way the user asks. Building on these results, we are developing a harness framework that reduces sycophancy, for example by breaking a large task into smaller ones.

We hope we can contribute our solutions and effort to the community, to researchers, and to developers, so that AI can serve people better.

Thank you again for your time and your support.

AuthorsOpenReview ↗

Summary of Revisions

Dear SAC, AC, and Reviewers,

Thank you for your careful reviews and the discussion. Your comments made the paper clearer and more accurate. We have uploaded the revised paper and summarize the changes below.

Title. The paper is now titled "When LLMs Are Sycophantic." The paper measures the conditions under which models yield and does not identify their causes. The introduction now states this directly.

Main changes.

  • Definitions. The introduction states both definitions of sycophancy and which one each claim uses. Personal choices are reported as preference endorsement. A new test compares each model's endorsement with its own advice before pressure (Section 4.7).
  • Verification cost. The min-k result is described as a certified structural association. Section 4.3 states the remaining confounds and adds evidence from linear systems and from checking tokens. Table 2 lists comparisons that hold more factors fixed.
  • Capability and reasoning. Section 4.4 reports all three capability groups. The maximum-reasoning result is stated over completed conversations and for the two models tested. Missing replies are listed by item.
  • Guardrails are defined by turn-1 behavior and checked on turn-1 replies.
  • Pressure. Results by turn and by tactic are reported in Section 4.6 and Appendix F, and pressure wording is described as adding little beyond repetition.
  • Practices. The Discussion matches practices to the type of question and marks which ones rest on a measurement.
  • Appendix F adds nine tables (S10 to S18). The main text remains within 10 pages.

Related work. We added 25 references.

  • Requested during the review. Nikeghbal et al. (2026), suggested by Reviewer AwEr, is discussed in Section 4.1. Least-to-most prompting, decomposed prompting, and chain-of-verification (Zhou et al., 2023; Khot et al., 2023; Dhuliawala et al., 2024) support the practice of splitting hard problems, as discussed with Reviewer s9u6. ELEPHANT (Cheng et al., 2026), cited in our reply to Reviewer v6Ub, now defines social sycophancy in the introduction and in Section 4.7.
  • Consistency across turns. Li et al. (2025a) evaluate the consistency of LLMs in sequential interactions, Li et al. (2025b) survey multi-turn interactions with LLMs, and Shnaidman (2026) shows that bias induced by user pressure can persist after the pressure is withdrawn. These works are cited in the related work and in the per-turn analysis of Section 4.6.
  • Other work. We also cite studies of answer stability under follow-up questions (Laban et al., 2023; Xie et al., 2024; Kim and Khashabi, 2025), reasoning models under multi-turn attacks (Li et al., 2026), and work by Cheng, Ibrahim, and colleagues on social sycophancy and its effects on users.

Why this matters. A single sycophancy rate hides when a model can be trusted. Our results show that yielding depends first on the task. Models rarely concede claims they can check cheaply, or answers they solve reliably. Where no answer can be checked, the choice of model matters most. These findings give users and developers concrete conditions to act on.

Open source and future work. We will release all items, transcripts, judge and human labels, and code, and we will contribute our evaluation code to the community. The supplementary package now also includes the unpressured answers behind the capability check. Next, we plan to build an open harness that reduces sycophancy in agent workflows. It will use these measurements to decide when to verify a claim, when to reason more, and which model to use.

Thank you again for your time and your thoughtful feedback.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.