- Rating
- 2
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper investigates when language models adopt a user's asserted answer or position under repeated conversational pressure. It evaluates eight models with reasoning disabled and two of them again at maximum reasoning, using 200 items, 13 pressure conditions, and four-turn conversations. Its main methodological contribution is a solver-certified, claim-specific measure of refutation size, min-k, evaluated through matched shallow/deep puzzle questions and complementary linear-system tasks. It also conditions adoption on independently estimated item-level capability. The authors report strong task-region differences, greater adoption on difficult-to-refute claims, higher adoption in low-capability strata, and no observed reasoning-item adoptions in the two maximum-reasoning configurations. Two LLM judges and a blinded human calibration support the reliability of the commitment labels.
Critical strengths
1.The design—using a solver to determine how many clues are needed to refute an incorrect answer—is quite interesting.
2.The evaluation is careful, with two judges, human annotation, and useful robustness checks
Critical weaknesses
1.Both options in the personal-choice tasks are reasonable. Supporting the option the user prefers is not necessarily sycophancy. This makes the high adoption rate in this category hard to interpret.
2.The shallow and deep questions share the clues, but ask about different answers. Puzzle size also changes across min-k levels. It is still unclear how much of the effect comes from verification cost rather than general task difficulty.
3.The capability analysis excludes the intermediate group, which accounts for about 60% of all deep-puzzle adoptions. The results show a difference between the extreme groups, but do not establish that most concessions are deference.
4.Maximum reasoning is tested on only two models, I think this is limited.
5.I do not believe this paper proposes a truly novel architecture to explain sycophancy in LLMs. Instead, it focuses on rather obvious factors—such as pressure, the specific model used, and the difficulty of verification—that clearly influence sycophantic behavior. In my view, the paper lacks innovation.
Questions for authors
- Does the region/model/pressure ordering remain, with comparable effect sizes, when reasonable personal-choice endorsement is excluded or analyzed separately? What evidence distinguishes a model's appropriate accommodation of a preference from the sycophancy interpretation used in the paper?
2.Please distinguish what the shallow/deep matching identifies from what remains confounded. If these effects cannot be separated, would you narrow the contribution to a certified structural association rather than an isolated verification mechanism?
3.The missingness is concentrated on difficult items. How should the zero-adoption result be interpreted over all attempted tasks?
- If the author insist I have some misunderstanding of your innovation, please clarify your innovation refer to my weakness.
6 comments
Response to Reviewer v6Ub (2/4): W2
W2: We agree. Within a pair, the deep question is also harder to solve, and min-k rises with puzzle size (r = 0.99). The matched pairs hold the puzzle, clues, topic, and answer format fixed, but they cannot separate checking cost from solving difficulty.
Separating the two requires tasks where they move apart. Linear systems do this. With more unknowns, a system becomes longer and more error-prone to solve by hand, yet a pushed answer is still refuted by one substitution. We measured both costs as the tokens a model spends to solve an item (turn 1) and to check the pushed answer (turn 2). Token counts require a maximum-reasoning run, and we use GPT-5.6-Luna because Claude-Sonnet-5's run is incomplete on the hardest items.
Table 1. GPT-5.6-Luna: median tokens at maximum reasoning, and adoption with reasoning off.
| Items | Solve (turn 1) | Check (turn 2) | Adoption |
|---|---|---|---|
| Common facts | 138 | 218 | 0.5% (3/648) |
| Shallow puzzles (min-k 1) | 125 | 344 | 0.0% (0/156) |
| Deep puzzles (min-k 4) | 334 | 490 | 3.6% (1/28) |
| Linear systems (4 or 6 unknowns) | 1,834 | 632 | 1.7% (4/233) |
| Linear systems (8 unknowns) | 8,273 | 719 | 0.0% (0/104) |
| Deep puzzles (min-k 8) | 5,820 | 2,781 | 18.8% (6/32) |
| Deep puzzles (min-k 12 or 16) | 32,826 | 10,335 | 13.5% (7/52) |
The key comparison is between eight-unknown systems and min-k 8 puzzles. The systems take more tokens to solve but far fewer to check, and they draw no adoption, while the puzzles draw 18.8%. Within linear systems, solving cost rises more than fourfold while checking cost barely changes, and adoption does not rise. Returning to deep puzzles, this model's adoption stops rising above min-k 8. With reasoning off, it solves only 24.4% of them, so beyond that depth it can neither solve nor check. The counts are also small (32 and 52 conversations). Pooled over all eight models, adoption rises from 6.4% at min-k 1 to 48.0% at min-k 12 or 16 (Section 4.3).
For this model, each doubling of checking tokens multiplies the odds of adoption by 2.1 (95% CI 1.8 to 2.5), while solving tokens add nothing (odds ratio 0.76, p = 0.2). The effect of checking tokens remains after controlling for the model's solve rate (odds ratio 2.1, CI 1.6 to 2.7). The same pattern holds beyond this model. Pooled over all eight models, using each model's own reply length in place of tokens gives the same result (odds ratio 2.03 per SD, solve rate controlled, CI 1.43 to 2.88).
Holding inability to solve fixed gives the same picture. Among items a model cannot solve (correct in at most 20% of its other conversations), adoption is 35.5% (114/321) on deep puzzles but 9.9% (141/1,420) on linear systems. Within models, unsolvable deep puzzles have 6.4 times the odds of adoption of unsolvable linear systems (95% CI 4.0 to 10.2).
Because puzzles and linear systems also differ in format, we treat these comparisons as supporting evidence rather than proof. A cleaner test stays within one task type. Across linear systems with 4, 6, and 8 unknowns, solving difficulty rises while checking stays at one substitution. Unlike for GPT-5.6-Luna alone, adoption pooled over all eight models rises from 2.1% at 4 unknowns to 14.4% at 8. Once we control for the model's solve rate on the item, system size no longer predicts adoption. So the ability to solve also matters, even when the check is cheap, and both factors contribute. The revised paper will report adoption and turn-1 solve rates by system size for each model.
Revision: We will remove "isolates," describe the min-k result as a certified structural association (see Q2), add this analysis to Section 4.3, and state the remaining confounds explicitly. We will keep min-k as our measure of verification cost, because it is certified by a solver and defined before any model is run. We will report checking tokens as a complementary, model-specific measure that agrees with it.
Response to Reviewer v6Ub (3/4): W3, W4, W5
W3: We agree. A difference in rates between the two extreme groups does not show that most concessions are deference. We counted all final-turn concessions on deep puzzles, with no group excluded. Groups follow Section 4.4. Solvable means the model solves the item in at least 80% of its other 12 conversations. Unsolvable means at most 20%, and intermediate is everything in between.
Table 2. Final-turn concessions on deep puzzles, eight models, reasoning off.
| Group | Adoption rate | Share of concessions |
|---|---|---|
| Solvable | 3.5% (11/310) | 3.6% |
| Intermediate | 44.8% (184/411) | 59.5% |
| Unsolvable | 35.5% (114/321) | 36.9% |
Unsolvable items account for 36.9% of concessions, not a majority, so we will no longer claim that most concessions are deference. What the data support is narrower: almost no concessions come from items the model solves reliably. Only 3.6% do; the other 96.4% come from items the model solves in fewer than 80% of its own conversations.
The intermediate group reflects unstable capability rather than knowledge the model holds. When a model in this group answered a deep puzzle correctly at turn 1, it still gave up that answer in 30.7% of conversations (46/150), close to the unsolvable group's rate.
Revision: We will report all three groups in Section 4.4, retitle it "Concessions come from items the model cannot reliably solve," and state the claim for deep puzzles only. We will make the same change in the abstract, the introduction, and Figure 1, replacing "hard items" with "deep puzzles."
W4: We agree that two models limit how far this result generalizes. Running at maximum reasoning is expensive. On the hardest puzzles, a single reply can need more than 30,000 tokens (median 32,826 for GPT-5.6-Luna at min-k 12 or 16). Some of Claude-Sonnet-5's replies exceeded our 64,000-token budget. Across all items and 13 conditions, each maximum-reasoning configuration therefore costs far more than a reasoning-off run, so we limited this comparison to two models from different developers.
The two models start far apart: with reasoning off, they solve 69.9% and 24.4% of deep puzzles. At maximum reasoning, both solve nearly all of the puzzles they complete. Neither concedes on any reasoning item where it completed its own turn-1 answer (0/548 and 0/648; see Q3). The result is consistent with Section 4.4: once a model can solve an item, it can check the pushed answer.
Revision: The revised paper states this result only for the two models tested and lists this limitation in the Discussion.
W5: Our contribution is a measurement framework for sycophancy. Each factor may seem intuitive on its own, but how much each one matters, relative to the others, had not been measured. We measure all four on the same items, turns, and judges, with a solver-certified measure of checking cost (min-k). Several results were not expected:
- Pressure wording adds little. On average, eight fallacies and four emotional framings yield no more adoption than plain repetition (12.9% and 13.2% vs. 15.0%). Only the appeal to popularity exceeds it, by 2 to 3 points.
- Checking cost matters beyond solving difficulty. Among items a model cannot solve, deep puzzles draw 35.5% adoption, while linear systems, which are cheap to check, draw 9.9% (W2).
- Concessions rarely come from items the model reliably solves (3.6% on deep puzzles, W3).
- A certified measure of checking cost. As you noted, min-k is computed and verified by a solver for each claim, which, to our knowledge, prior benchmarks do not provide.
These results are also practical: they suggest choosing models by their measured profile, splitting problems into steps that are cheap to check, and asking without a preferred answer.
Revision: We will state these contributions explicitly in the introduction, and release all items, transcripts, labels, and code.
Response to Reviewer v6Ub (1/4): W1
Thank you for your careful review and your detailed questions. We are also glad that you found the solver-based design interesting and the evaluation careful. We answer each weakness (W1 to W5) and then each question (Q1 to Q4), in four comments. All analyses use the runs reported in the paper.
W1: We agree that endorsing a reasonable option the user prefers is not an error in itself. We will report this region as preference endorsement, separate from error adoption.
We included this region deliberately, as one end of a spectrum. At one end are questions with a checkable answer, where agreeing with a wrong claim is clearly an error. In the middle are questions with normative weight, such as contested policy and moral scenarios. At the other end are personal choices, where no option is wrong and nothing in the question resists agreement. This end shows how strong the tendency to agree with the user is when nothing holds it back. Comparing it with the other regions shows how much a checkable answer or a norm reduces that tendency. It also reflects a common use. People often ask models for advice on everyday decisions where no shared standard exists and judgments vary from person to person. This is where agreement is hardest to check and most likely to go unnoticed. The two analyses requested in Q1 address the interpretation directly (see Q1).
Revision: We will report personal choices as preference endorsement and add both analyses from Q1. In the Discussion, we will explain why we include this region: it is one end of the spectrum, and it matters for everyday decisions where no shared standard exists.