- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FIGS, a benchmark for evaluating multi-turn sycophancy while distinguishing it from appropriate emotional acknowledgment. Its taxonomy separates two axes: Sycophancy, covering evidence-sensitive judgment, proportionality, and resistance to unsupported pressure; and Calibrated Validation, covering acknowledgment of user disclosures, faithful representation of their experience, and respect for their choices.
The benchmark contains 500 frozen scenarios across six domains and three severity levels. An adaptive user simulator conducts five user–assistant exchanges, and an automated judge scores each complete transcript on both axes using a four-level severity scale and ten diagnostic rules.
Experiments evaluate eight models under baseline and factual system prompts, with additional calibrated-prompt experiments on four models. The paper finds that the factual prompts reduce sycophancy but can increase validation failures, while a prompt addressing both objectives improves their balance. Additional analyses examine human–judge agreement, alternative simulators, rollout variance, prompt variants, and comparisons with prior evaluation protocols.
Critical strengths
-
The benchmark separates two consequential dimensions that can otherwise be conflated. Distinguishing unsupported agreement from grounded acknowledgment allows an evaluator to identify both sycophantic warmth and factually careful but dismissive responses. Including warranted updating further avoids treating every change of position as a failure. This makes FIGS useful for assessing interventions that might improve an anti-sycophancy score by degrading conversational support.
-
The evaluation framework is supported by substantial methodological documentation and robustness analyses. The paper provides 500 scenarios, adaptive simulation, explicit scoring rubrics, human–judge comparisons, alternative simulator experiments, rollout reruns, and prompt ablations. The calibrated-prompt experiment is particularly useful because it demonstrates that the two axes need not move in opposite directions. The authors also appropriately disclose that this prompt was tuned against their rubric rather than presenting it as an independently validated general solution.
Critical weaknesses
-
The prompt experiments do not isolate whether prioritizing accuracy itself causes validation failures. The baseline explicitly encourages a warm, emotionally responsive persona. The factual prompt also changes tone, response structure, and the permitted treatment of emotional disclosures. The additional minimal and prose prompts provide useful checks, but both remove the baseline persona and request short replies, as Appendix F.3 acknowledges. Consequently, the observed changes may partly reflect altered style instructions rather than an intrinsic cost of resisting sycophancy. This matters because the “paternalism trap” is a central empirical claim. A more controlled comparison should preserve warmth, length, and structure while varying only the evidentiary and anti-capitulation instructions.
-
Validation of the primary pass/fail metric remains incomplete. The headline metric treats any score of 2 or higher as failure, yet the reliability analysis primarily reports four-level quadratic-weighted kappa and within-one agreement. These measures can remain favorable when disagreements cross the critical boundary between scores 1 and 2. This is especially relevant because 93% of residual sycophancy failures under the factual prompt receive score 2. Human exact-score agreement is modest, and the held-out judge test uses a single annotator rather than an independent panel. Binary confusion matrices and independently adjudicated labels would strengthen confidence in failure rates and close model comparisons. The absence of observed S1.b failures also leaves sensitivity to warranted-update failures less established than sensitivity to capitulation.
Questions for authors
The following clarification and validation requests could materially affect my assessment:
-
Does the increase in calibrated-validation failures persist when the baseline’s warmth, response length, and persona are preserved and only the anti-sycophancy instructions are added? The existing prompt ablations are useful but still vary these factors together.
-
What are human–human and judge–human agreement at the binary pass/fail boundary, including false-positive and false-negative rates? Do the main model rankings and prompt effects persist with human-adjudicated labels and with only scores 3–4 counted as failures?
-
Can you validate sensitivity to S1.b failures using a balanced set of warranted updates and unwarranted reversals, independently labeled by multiple annotators? The constructed example demonstrates that the judge can assign the label, but does not establish detection reliability.
-
The temporal analysis reports where the judge cites a failure rather than independently labeling every assistant reply. Does the conclusion that sycophancy builds over turns hold when measuring the first failure onset or per-reply failure rates, accounting for the changing opportunities for pressure across turns?
5 comments
*(Part 2 of 4)*
(Part 2 of 4)
1. W1 and Q1 (continued)
What the comparisons show.
- Resisting sycophancy alone does not raise CV failures for three of the four assistants (baseline vs persona-controlled). With the baseline kept whole, CV stays at the baseline level for Gemini, DeepSeek and Sol (6→9, 6→3, 1→3) while sycophancy still falls by 26–54 points.
- Replacing the baseline persona does (persona-controlled vs the other three). Every prompt that replaces it raises CV failures for Gemini (32–40%) and Sol (16–27%).
- It is not the factual prompt's wording about emotions (factual vs minimal). The minimal prompt says nothing about emotions, yet its CV rise is as large as the factual prompt's.
- Permitting warmth softens the cost but does not remove it (prose vs minimal). Allowing warmth that matches the user lowers CV failures (Gemini 40→32%, Sol 27→16%, Grok 50→28%), but for Gemini and Sol they stay well above the baseline.
- Grok is the exception. Its CV failures rise to the factual prompt's level under the resistance instructions alone (28→39%, against 38% under the factual prompt), so for Grok the cost does come from resisting sycophancy. The judge's descriptions show it applying the evidentiary rules to the user's feelings, for example a reply that "converts the disclosure into evidentiary items with zero uptake of the fear".
The persona-controlled prompt leaves more sycophancy than the factual prompt (9–21% against 0–10%).
Evidence without any instruction: activation steering (new Section 5.2, Figure 5, Appendix H).
To test resisting sycophancy with no style instruction at all, we steer four open-weight models along a sycophancy direction in the residual stream, with no system-prompt change. The direction comes from 115 sycophancy scenarios that are excluded from all results, and no CV data is used to build it or to choose its layer or strength. k is the steering strength (k = 0 is the unsteered model; negative k pushes away from sycophancy). Bold changes are significant against k = 0 on the same scenarios (275 held-out sycophancy and all 110 CV scenarios).
| Model | Sycophancy fail %, k = 0 / −4 / −8 | Change at k = −4 | Change at k = −8 | CV fail %, k = 0 / −4 / −8 | Change at k = −4 | Change at k = −8 |
|---|---|---|---|---|---|---|
| Gemma-4-26B-A4B | 80.3 / 60.4 / 38.0 | −20.1 | −42.3 | 9.1 / 10.0 / 24.5 | +0.9 | +15.5 |
| Mistral-Small-24B | 88.0 / 72.6 / 44.2 | −15.3 | −43.8 | 46.4 / 52.7 / 72.7 | +6.4 | +26.4 |
| OLMo-3.1-32B | 88.7 / 67.9 / 35.8 | −20.8 | −52.9 | 13.6 / 14.5 / 38.2 | +0.9 | +24.5 |
| Qwen3-8B | 96.4 / 81.8 / 47.1 | −14.6 | −49.3 | 37.6 / 34.5 / 70.0 | −2.8 | +33.0 |
Pushed hard enough (k = −8), every model becomes markedly less sycophantic and, at the same time, markedly worse at calibrated validation, although nothing in the direction targets validation. A moderate push (k = −4) separates the two: sycophancy falls by 15–21 points while no model's CV rate changes significantly. Full results, including k = +4, are in Appendix H.
The two experiments answer the question from opposite sides, and they point the same way: pushing sycophancy to its lowest levels is what brings on the paternalism trap. The persona-controlled prompt removes most sycophancy with no validation cost for three of the four assistants, but it leaves more sycophancy than the factual prompt (9–21% against 0–10%). The factual prompt reaches the lowest sycophancy of all, and that is where validation pays (CV 26–39% for Gemini and Sol). Steering shows the same with no instruction at all: a moderate push leaves validation intact, and only a hard push, which drives sycophancy lowest, degrades it in every model. So the trap is not an intrinsic cost of prioritizing accuracy, but a risk that appears as a mitigation pushes sycophancy furthest down, to a degree that depends on the model and the intervention (Grok shows it even under the persona-controlled prompt). We have added this clarification to the revised paper: Section 5.1 now shows that the factual prompt's CV cost appears where sycophancy is pushed lowest, with the persona-controlled and steering results, and the abstract and conclusion now state that the size of the trade-off depends on the model and the mitigation. The dual-axis argument is unaffected: a sycophancy-only score would rank the factual prompt above the persona-controlled one (10% vs 21% for Gemini), while FIGS shows it costs 30 more points of CV failures.
*(Part 3 of 4)*
(Part 3 of 4)
2. W2 (first part) and Q2
To make agreement at the 1|2 boundary visible directly, rather than through QWK and within-one agreement, we now validate the binary pass/fail decision at both cutoffs (score > 1 and 3–4), in the revised Appendix D (D.3, D.4, D.6).
Enlarged held-out test set. We enlarged the test set from 44 to 164 transcripts (124 sycophancy and 83 CV labels), none used in judge development. Of the 120 added transcripts, 93 are drawn from the main evaluation (all 8 assistants) and 27 from other assistants to cover rare rules; 50 are double-annotated.
Judge–human agreement at the boundary (FPR: judge fails a human pass; FNR: judge passes a human failure):
| Axis, cutoff | Validation (n = 50) | Test (n = 124 sycophancy / 83 CV) | FPR / FNR (test) |
|---|---|---|---|
| Sycophancy, > 1 | 80% | 88% | 8/34 / 7/90 (8%) |
| Sycophancy, 3–4 | 94% | 77% | 10/54 / 18/70 |
| CV, > 1 | 82% | 89% | 3/37 / 6/46 (13%) |
| CV, 3–4 | 86% | 81% | 9/61 / 7/22 |
Binary confusion matrices at > 1 (both fail / judge-only fail / human-only fail / both pass):
| Axis | Validation | Test |
|---|---|---|
| Sycophancy | 19 / 9 / 1 / 21 | 83 / 8 / 7 / 26 |
| CV | 14 / 0 / 9 / 27 | 40 / 3 / 6 / 34 |
The judge rarely misses a failure a human sees (FNR 8% and 13% on the test set). The transcripts added to the test set are deliberately enriched for severe failures, so human passes, the FPR denominators, are relatively few. On the score scale, the judge reaches QWK 0.66 (sycophancy) and 0.78 (CV) on the test set.
Human–human agreement (50 double-annotated test transcripts). Agreement is 81–84% at > 1 and 84% at 3–4, with QWK 0.58 (sycophancy) and 0.66 (CV), matching the published 3-annotator panel (0.60 / 0.61). On the validation set, the judge agrees with the panel as often as an individual annotator does. Exact-score agreement is modest for humans too (53–55%). The independent second judge agrees with the main judge at QWK 0.77 on both axes of the test set.
Where the judge scores 2, humans also fail the conversation in 90% (26/29, sycophancy) and 84% (16/19, CV) of test cases, and in 57% (12/21) and 6/6 of validation cases.
Do rankings and prompt effects persist under human-adjudicated labels and at the 3–4 cutoff? We map each run's judge-score distribution through P(human failure | judge score), estimated from 147 (sycophancy) and 133 (CV) human-labelled transcripts.
- No significantly ordered pair of models reverses, at either cutoff (48 sycophancy pairs at both; 28 and 30 CV pairs).
- All 12 sycophancy prompt effects remain significant with unchanged sign at both cutoffs. Gemini's baseline-to-factual drop, for example, is −57 points from judge labels, −38 human-calibrated, and −27 at 3–4. So the factual-prompt reduction does not rest on the score-2 boundary.
- CV: 7 of 12 prompt effects remain significant at > 1 and 5 at 3–4, in line with 5–6 under the published labels; every sign is unchanged at > 1.
At the boundary that every failure rate depends on, the judge matches humans on 88–89% of held-out decisions, more often than two humans match each other (81–84%).
*(Part 1 of 4)*
(Part 1 of 4)
We thank the reviewer for the thoughtful review and for recognizing several strengths of FIGS. It separates two consequential dimensions, so it can identify both sycophantic warmth and factually careful but dismissive responses, without treating every change of position as a failure; this makes it useful for assessing interventions "that might improve an anti-sycophancy score by degrading conversational support". It is supported by substantial documentation and robustness analyses, and the calibrated-prompt experiment shows that the two axes need not necessarily move in opposite directions.
We are also grateful that the reviewer named the analyses that could change the assessment. We ran each of them.
Summary.
A new prompt that keeps the baseline persona and adds only the anti-sycophancy instructions removes most sycophancy with little validation cost for three of four assistants, while the factual prompt, which drives sycophancy lowest, has the largest cost; activation steering shows the same pattern without any instruction. We have revised the paper to say that the paternalism trap appears when sycophancy is pushed to its lowest levels.
As requested, we now report human–human and judge–human agreement at the binary pass/fail boundary, with binary confusion matrices and false-positive and false-negative rates, at both the > 1 and 3–4 cutoffs, on an enlarged test set of 164 transcripts. The judge agrees with humans on 88–89% of pass/fail decisions, at or above human–human agreement, and rankings and prompt effects hold under human-calibrated labels and at the 3–4 cutoff.
We also clarify how S1.b failures reach the rule stage, and report a balanced minimal-pair set on which the rule judge detects failures to update in both directions and never tags a warranted update. Finally, we report first-failure onset and per-reply failure rates from labelling every reply independently: 70% of sycophancy failures first appear after the first reply.
Below we answer each concern in turn, taking related concerns together.
1. W1 and Q1
As the reviewer suggested, we added a prompt that keeps the baseline's warmth, length and persona and adds only the anti-sycophancy instructions, and we also tested activation steering with no instruction at all. CV failures do not rise for three of the four assistants; for Grok 4.6 they do.
What each accuracy prompt changes. Every accuracy prompt adds instructions to resist sycophancy. They differ in what else they change about the baseline (the warm assistant persona of the main paper): whether they replace its persona and warmth, and what they say about emotional disclosures. Comparing them therefore separates the effect of resisting sycophancy from the effect of everything else:
The persona-controlled prompt is the complete baseline prompt followed by five sections copied verbatim from the factual prompt. Nothing from the factual prompt that governs tone or emotional disclosures is included. All five prompts were run on the same 200 scenarios (100 per axis) with the same four assistants, simulator and judge, and compared with the baseline on paired scenarios.
Results. All four prompts cut sycophancy significantly for every assistant:
They differ in what they cost on calibrated validation (bold: significant increase over the baseline):
† Not significant on the 100 CV scenarios alone, but significant when CV is scored on all 200 conversations, and when only scores 3–4 count as failures. No unbolded change is significant.