acceptodds
Under review as a conference paper at ICLR 2027

When Pleasing the User Compromises Care: AI Sycophancy in Caregiver-Mediated Healthcare

Abstract

Large language model (LLM) alignment often assumes that the user interacting with a model is also the primary beneficiary of its behavior. Yet in many consequential settings, one person directs the system while another bears the consequences, creating a potential conflict between accommodating the user and protecting the beneficiary. We study this problem in caregiver-mediated healthcare, where parents, pet owners, and caregivers make decisions for children, animals, and older adults. We focus on third-party sycophancy: whether an AI advisor shifts toward a caregiver’s preferred but clinically inferior action at the expense of appropriate care for the patient. We introduce an evidence-grounded benchmark spanning veterinary, pediatric, and older-adult care. Holding clinical facts fixed, we vary caregivers’ stated preferences and communication styles and evaluate both initial recommendations and multi-turn follow-up. Across frontier and open-weight models, stated inferior preferences cause substantial additional switching from appropriate to inappropriate care beyond failures observed during neutral follow-up, while communication style alone has a smaller effect. These failures accumulate across conversational turns, including on cases models initially answer correctly, and often carry moderate or severe potential harm. These results highlight a broader alignment challenge: when the user and beneficiary differ, accommodating user preferences can come at the expense of the party who bears the consequences.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Healthcare AI, Sycophancy, AI Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ytNUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CareDistance, a benchmark for studying third-party sycophancy in caregiver-mediated healthcare, where the person interacting with an AI system is distinct from the patient who bears the consequences of its advice. The benchmark contains 450 source-grounded cases across eldercare, pediatrics, and veterinary care. For each fixed clinical case, the authors vary caregiver preferences and five communication personas, yielding 13 experimental conditions, and evaluate advisor behavior over conversations of up to five turns.

Critical strengths
  1. The paper is well organized, and the experimental logic is easy to reconstruct.
  2. The paper does not simply compare a neutral question with an adversarial prompt. Patient facts are kept fixed, caregiver preference and communication style are factorially varied, neutral follow-up provides a multi-turn control, and a gold-preference condition provides an additional sanity check. The authors also explicitly prevent the caregiver simulator from introducing new clinical facts or objective constraints during follow-up. This is consequential because it makes the observed preference-associated differences substantially more interpretable than an ordinary adversarial-prompt benchmark.
  3. CareDistance spans three beneficiary-dependent healthcare settings, five decision categories, 450 clinical cases, multiple model families, and conversations extending to five turns. The resulting effect is not confined to one model: inferior preference increases switching beyond neutral follow-up across all six models evaluated in all three domains, with large effect sizes for several systems. The source-grounded formulation and explicit separation of action agreement, overall care appropriateness, and potential harm also make the benchmark more clinically meaningful than purely stance-based sycophancy metrics.
Critical weaknesses
  1. The paper convincingly demonstrates that caregiver pressure can shift models toward clinically inferior recommendations. However, it does not include the natural counterfactual in which the same clinical facts and same inferior preference are expressed by the patient directly rather than by a caregiver. Therefore, the experiments establish sycophancy in a user–beneficiary-separated setting, but do not establish whether user–beneficiary separation changes susceptibility relative to ordinary first-party medical sycophancy. This matters because closely related multi-turn medical sycophancy has already been demonstrated in prior work. A relatively small matched first-party experiment on the human-care domains could substantially sharpen the novelty claim. Alternatively, the authors should frame user–beneficiary separation as an important application/measurement setting rather than implying that the experiments identify a separation-specific mechanism.

  2. The 450 cases, reference actions, and inferior alternatives are generated and verified primarily through LLM-based procedures grounded in clinical sources. The main experimental labels are then produced by one GPT-6 Luna judge. Human validation is restricted to 60 conversations from 30 automatically stratified pairs, two medical trainees, and no veterinary cases; additionally, the human estimates indicate substantially more neutral degradation than the automatic judge detects. This does not invalidate the qualitative finding, but the current evidence is weaker for precise claims about the magnitude of switching and especially the reported distribution of moderate/severe clinical harm. Larger independent expert validation—including veterinarians and validation of case/reference labels and harm severity—would materially strengthen the paper.

Questions for authors
  1. Can the authors test a matched control where the patient is the user, at least for a representative small set of eldercare and pediatric cases? In this control, the patient states the same worse preference. All other clinical facts are the same. If the result is similar to the caregiver condition, I would see the main contribution mainly as a clinical extension of existing medical sycophancy work. If separation clearly changes behavior, this would greatly strengthen the main claim.

  2. How much do the conclusions depend on the choice of automatic judge and the information it gets? The judge is given the separate neutral first response, the conversation being evaluated, and the source evidence.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ytNU (Part 1/2)

Thank you for your thoughtful review and for recognizing the value of our experimental controls and the distinction between clinical appropriateness and action agreement. If these responses resolve your concerns, we would appreciate your reconsideration of the score.

Matched patient–caregiver comparison (Weakness 1 and Q1)

Can the authors evaluate a matched first-party control, at least for a representative subset of eldercare/pediatric cases, in which the patient expresses the same inferior preference with otherwise identical clinical facts?

Following your suggestion, we ran the matched patient–caregiver comparison with the same clinical facts and inferior action, and found no consistent difference in susceptibility. CareDistance studies a general alignment question in multi-stakeholder interactions: whether LLMs account for a separate beneficiary’s interests when the user prefers a conflicting course. In caregiver-mediated healthcare, several plans may be clinically appropriate, and choosing among them requires considering stakeholders’ roles, preferences and who bears the consequences [1,2]. Preserving factually correct answers, the focus of much work on mitigating LLM sycophancy [3,4], does not by itself resolve how an LLM should respond when these interests conflict. We evaluate clinical appropriateness separately from action agreement to distinguish acceptable accommodation from advice that compromises the beneficiary’s care.

In selected caregiver cases where patients could plausibly participate, LLMs often gave inappropriate advice without seeking their wishes. We then tested whether explicit beneficiary input guides advice when preferences conflict. Keeping the caregiver’s inferior request unchanged, we varied whether the patient or another relative wanted the same recommended care. Stating the patient’s wishes reduced failures more than stating another relative’s preference, yet LLMs still gave clinically inappropriate advice in 17–41% of conversations across models despite explicitly knowing that the patient wanted recommended care.

These results identify an alignment failure: LLMs do not reliably account for the beneficiary’s role and interests when advising the user (caregiver), despite showing some sensitivity to whose preference is stated. First-party sycophancy evaluations alone cannot establish whether LLMs account for a separate beneficiary’s interests when that beneficiary’s wishes are unknown or conflict with the user’s preferences. Our controlled comparisons of stakeholder roles and preferences, alongside clinical evaluation of the resulting advice, provide evidence to guide alignment training and reward modeling for multi-stakeholder interactions. The shared response (Part 1/3, Part 2/3, Part 3/3) presents these experiments, results and alignment implications.

References

[1] Sorensen et al. Position: A Roadmap to Pluralistic Alignment. ICML 2024.

[2] Gabriel and Keeling. A matter of principle? AI alignment as the fair treatment of claims. Philosophical Studies, 2025.

[3] Chen et al. From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning. ICML 2024.

[4] Li et al. Causally Motivated Sycophancy Mitigation for Large Language Models. ICLR 2025.

AuthorsOpenReview ↗

Response to Reviewer ytNU (Part 2/2)

Clinical validation and potential harm (Weakness 2)

Larger independent expert validation—including veterinarians and validation of case/reference labels and harm severity—would materially strengthen the paper.

We extended the original pediatric and eldercare conversation review to veterinary care and added clinical assessments of the benchmark cases. All 32 reviewed cases were judged well grounded in their cited clinical sources. A veterinarian (DVM) also judged 14/15 reference actions appropriate and 14/15 inferior actions clinically less appropriate.

For veterinary response validation, the veterinarian reviewed 30 five-turn conversations with model identities and previous judgments hidden. Agreement with GPT-6 Luna was 96.7% (κ = 0.934) on the first clinically inappropriate turn or on care remaining appropriate throughout.

To examine the effect-size discrepancy, an additional medical trainee adjudicated 19 disputed conversations and five agreement controls. The discrepancy mainly concerned additional failures identified by human reviewers in neutral conversations. Incorporating the adjudicated decisions gives an excess of 30.0 percentage points under inferior preference compared with neutral follow-up (95% CI: 16.7–43.3), versus 46.7 points under the original LLM judge. The new veterinary review found a corresponding excess of 33.3 points (95% CI: 20.0–46.7).

The additional case assessments and human–LLM agreement results across all three domains, together with adjudication of disputed judgments, strengthen confidence in the benchmark’s clinical grounding and evaluation, and support a substantial increase in inappropriate care under inferior caregiver preferences. We agree that precise effect sizes and the distribution of harm severity require broader clinical validation. Our shared response on clinical validation provides the full assessments and comparisons.

Sensitivity to the judge and neutral reference (Q2)

How sensitive are the conclusions to the choice and information available to the automatic judge? The judge is explicitly given the separate neutral first response in addition to the evaluated conversation and source evidence.

We tested whether the increase in inappropriate care under inferior preference depends on the judge or its access to the separate neutral reference. We used saved neutral and inferior-preference conversations for Claude Sonnet 5, GPT-5.6 Sol, GPT-5.6 Luna and Gemini 3.8 Flash, with 120 questions per advisor (40 each in eldercare, pediatrics and veterinary care).

Using the same conversations, clinical evidence and rubric, we compared:

  • Repeated judging: GPT-6 Luna with the original full input.

  • Without neutral reference: GPT-6 Luna, removing only the separate neutral first exchange.

  • Different judge: Claude Sonnet 5 with the original full input.

Increase in inappropriate care under inferior preference

Advisor modelOriginal GPT-6 LunaRepeat GPT-6 LunaGPT-6 Luna without neutral referenceClaude Sonnet 5 (full original input)
Claude Sonnet 553.3 [43.1–63.1]51.7 [41.0–62.1]51.7 [41.2–61.7]59.0 [47.6–68.5]
GPT-5.6 Sol29.2 [20.6–38.1]25.0 [16.9–33.7]32.5 [24.1–40.9]34.2 [25.0–43.3]
GPT-5.6 Luna28.3 [19.7–37.1]33.3 [24.8–42.1]30.0 [21.6–38.7]31.7 [21.1–42.5]
Gemini 3.8 Flash26.7 [16.8–36.4]25.8 [14.7–36.6]21.7 [10.5–32.8]35.0 [25.4–44.7]

Inferior preference minus neutral follow-up, in percentage points (95% CIs).

The increase in inappropriate care under inferior preference remains consistent across judges and settings, with only modest differences in estimated magnitude.

Dear Reviewer ytNU,

As the discussion period is close to coming to an end, we wanted to follow up and ask whether our responses and revised manuscript have addressed the concerns you raised. We would be happy to clarify or expand on any remaining points. We sincerely appreciate your time and thoughtful feedback, which have helped us strengthen the evaluation and clarify the paper’s contribution.

Reviewer_tS6mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is investigating the third-party sycophancy in LLMs. It studies a failure mode in which an AI advisor can have a tendency of accommodating the preference of an interacting user; this problem has been studied in the caregiver-medicated healthcare. The paper has introduced the CareDistance benchmark containing 450 clinically grounded cases across pediatric, eldercare and veterinary scenarios. Also, they have evaluated several LLMs with different caregiver preferences and communication styles.

The results indicate that models often start with appropriate medical advice but has the tendency to change their recommendations when caregivers can repeatedly ask for less optimal treatments. The authors also compare these results against normal conversations without the pressure of the caregiver in order to separate the effects of user preferences from the general conversational errors.

The paper concludes that caregiver preferences can have a much stronger effect than communication style alone. The authors, in this study, have also explored if adding instructions to prioritize the patient’s well-being can reduce these failures

Critical strengths

Important problem with a useful benchmark

The paper has addressed a critical problem that has not received enough attention in the research. The problem is real that in many real-world situations, the person interacting with an AI system is not the person affected by its recommendations. This can be extremely important or significant in the healthcare sector, where incorrect advice can have serious consequences.

I think the CareDistance benchmark can be considered useful contribution. It covers 450 cases across three healthcare domains and tests different caregiver preferences and communication styles. This provides a good starting point for studying this problem.

Good experimental design with multi-turn evaluation

I like the idea that the authors have evaluated the models over multiple conversation turns instead of looking only at the first response.

I believe that the comparison with neutral conversations is also a strength of this work. I think it can possibly help separate errors caused by the caregiver preferences from the errors that happen naturally during longer conversations.

The results shown in this work indicate that even models that initially provide correct or optimal medical advice can change their recommendations or preferences after repeated requests. I think this is an important and critical finding that also highlights the limitations of single-turn evaluations.

Critical weaknesses

Human validation is limited

The biggest concern, in my opinion, is that the paper relies heavily on an LLM judge to determine whether medical recommendations are optimal or appropriate. The authors have validated the results by using only 60 conversations which are actually reviewed by only two medical trainees. This is a relatively small sample compared with the full benchmark, and even in that review, no veterinary cases were included.

I noticed that Table 8 shows a large difference between the LLM judge based and human based evaluations. The LLM judge is reporting an excess switching rate of 46.7 percentage points. On the other hand, the two human reviewers report only 20.0 and 16.7 percentage points.

Unclear if the findings are specific to third-party sycophancy.

The paper has focused on situations where the caregivers are making decisions for someone else. However, previous studies (Kim et al. (2026) : The Doctor Will Agree With You Now: Sycophancy of Large Language Models in Multi-Turn Medical Conversations) have already shown that LLMs may change their medical recommendations when the users repeatedly push for a certain answer.

The authors have not provided any comparison of caregiver-mediated conversations with similar conversations where patients directly ask for the similar treatment.

Therefore, I think, it is difficult to tell whether the observed behavior is a new problem related to third-party interactions or simply another example of general LLM sycophancy.

Questions for authors

Q1: Can the authors explain the difference in table 8 between llm judge and human reviewers in more detail? How confident are they that the main results would remain similar with a larger human evaluation?

Q2: Any of the 450 cases independently reviewed by medical professionals?

Q3: Have the authors tested if the results remain consistent when the same experiments are repeated with different generated conversations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer tS6m (Part 1/2)

Thank you for your thoughtful review and for recognizing the value of the benchmark and its neutral multi-turn control. If these responses resolve your concerns, we would appreciate your reconsideration of the score.

Human–LLM differences and effect size (Q1)

Can the authors explain the difference in table 8 between llm judge and human reviewers in more detail? How confident are they that the main results would remain similar with a larger human evaluation?

The difference in Table 8 mainly reflects human reviewers identifying more failures in neutral conversations. To examine these disagreements, an additional medical trainee adjudicated the 19 disputed conversations and five agreement controls, agreeing with the LLM judge in 12 of the 19 disputes. Updating the original study with these decisions gives inappropriate-care rates of 20% under neutral follow-up and 50% under inferior preference, an excess of 30 percentage points (95% CI: 16.7–43.3), compared with 46.7 percentage points under the LLM judge. The new veterinary review also found an excess of 33.3 percentage points (95% CI: 20.0–46.7). Together, these clinical assessments support a substantial increase in inappropriate care under inferior caregiver preferences, strengthening our confidence that the main finding would persist in a larger human evaluation, while broader clinical review would help estimate the effect size more precisely. Our shared response on clinical validation provides the full comparison.

Independent clinical validation (Weakness 1 and Q2)

Any of the 450 cases independently reviewed by medical professionals?

Yes. We added clinical assessment of the benchmark cases themselves and extended the original conversation review to veterinary care. For the conversation review, a veterinarian (DVM) assessed 30 five-turn conversations from 15 cases. As in the original pediatric and eldercare review, model identities and previous judgments were hidden. Agreement with GPT-6 Luna was 96.7% (κ = 0.934) on the first clinically inappropriate turn or on care remaining appropriate throughout.

The case assessments covered these 15 veterinary cases and 17 pediatric and eldercare cases; all 32 were judged well grounded in their cited clinical sources. The veterinary clinician additionally judged 14/15 reference actions appropriate and 14/15 inferior actions clinically less appropriate. Our shared response on clinical validation provides the detailed assessments and results.

Consistency across newly generated conversations (Q3)

Have the authors tested if the results remain consistent when the same experiments are repeated with different generated conversations?

Yes. We repeated the five-turn neutral and inferior-preference conversations for Claude Sonnet 5, GPT-5.6 Sol, GPT-5.6 Luna and Gemini 3.8 Flash on 120 questions per model (40 each in eldercare, pediatrics and veterinary care). We generated new advisor replies and adaptive caregiver follow-ups using the original prompts and settings, then evaluated them with the original GPT-6 Luna judge. The increase under inferior preference recurred in all four models.

Increase in inappropriate care under inferior preference

Advisor modelOriginal runNew runChange (new − original)
Claude Sonnet 553.3 [43.1–63.1]55.0 [45.4–64.3]+1.7 [−10.6–13.9]
GPT-5.6 Sol29.2 [20.6–38.1]29.2 [20.3–37.9]0.0 [−8.9–9.0]
GPT-5.6 Luna28.3 [19.7–37.1]20.0 [11.2–28.7]−8.3 [−18.0–1.3]
Gemini 3.8 Flash26.7 [16.8–36.4]25.0 [14.3–35.6]−1.7 [−13.3–10.6]

Inferior preference minus neutral follow-up, in percentage points (95% CIs). Change is new minus original.

The increase over neutral follow-up was 20.0–55.0 percentage points in the new run. The paired 95% CIs for changes between runs included zero, with no statistically clear change for any model.

AuthorsOpenReview ↗

Response to Reviewer tS6m (Part 2/2)

Patient–caregiver comparison and prior medical sycophancy work (Weakness 2)

Therefore, I think, it is difficult to tell whether the observed behavior is a new problem related to third-party interactions or simply another example of general LLM sycophancy.

We agree that prior work has established medical sycophancy by measuring LLMs resistance to repeated user pushback on diagnostic and biomedical questions [1]. Much of the work on mitigating LLM sycophancy focuses on preserving factually correct answers when users express conflicting views [2,3]. In multi-stakeholder care decisions, several plans may be clinically appropriate, and choosing among them requires considering stakeholders’ roles, preferences and who bears the consequences [4,5]. Factual correctness alone does not resolve how an LLM should respond when these interests conflict. CareDistance examines this general alignment question in caregiver-mediated healthcare: whether LLMs account for a separate beneficiary’s interests, including when that beneficiary is absent from the conversation. We evaluate clinical appropriateness separately from agreement with the reference or inferior action, allowing us to distinguish acceptable accommodation from advice that compromises the beneficiary’s care.

Following your suggestion, we ran the matched patient–caregiver comparison with the same clinical facts and inferior action, and found no consistent difference in susceptibility. In caregiver cases where patients could plausibly participate, LLMs often gave clinically inappropriate advice without seeking their wishes. We then tested whether explicit beneficiary input guides advice when preferences conflict. Keeping the caregiver’s inferior request unchanged, we varied whether the patient or another relative wanted the same recommended care. Stating the patient’s wishes reduced failures more than stating another relative’s preference, yet LLMs still gave clinically inappropriate advice in 17–41% of conversations across models despite explicitly knowing that the patient wanted recommended care.

These results identify an alignment failure: LLMs do not reliably account for the beneficiary’s role and interests when advising the user (caregiver), despite showing some sensitivity to whose preference is stated. First-party sycophancy evaluations alone cannot establish whether LLMs account for a separate beneficiary’s interests when that beneficiary’s wishes are unknown or conflict with the user’s preferences. Our controlled comparisons of stakeholder roles and preferences, alongside clinical evaluation of the resulting advice, provide evidence to guide alignment training and reward modeling for multi-stakeholder interactions. Our shared response (Part 1/3, Part 2/3, Part 3/3) details these controls, results and alignment implications.

References

[1] Kim et al. The Doctor Will Agree With You Now: Sycophancy of Large Language Models in Multi-Turn Medical Conversations. 1st Workshop on Linguistic Analysis for Health (HeaLing 2026).

[2] Chen et al. From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning. ICML 2024.

[3] Li et al. Causally Motivated Sycophancy Mitigation for Large Language Models. ICLR 2025.

[4] Sorensen et al. Position: A Roadmap to Pluralistic Alignment. ICML 2024.

[5] Gabriel and Keeling. A matter of principle? AI alignment as the fair treatment of claims. Philosophical Studies, 2025.

Dear Reviewer tS6m,

As the discussion period is close to coming to an end, we wanted to follow up and ask whether our responses and revised manuscript have addressed the concerns you raised. We would be happy to clarify or expand on any remaining points. We sincerely appreciate your time and thoughtful feedback, which have helped us strengthen the evaluation and clarify the paper’s contribution.

Reviewer_ZzS7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CareDistance, a benchmark of 450 cases across veterinary, pediatric, and older-adult care. It evaluates whether AI advisors provide inappropriate advice when caregivers express preferences that conflict with the patient’s clinical interests.

Critical strengths

The paper addresses a practical concern: caregivers’ preferences may conflict with patients’ clinical interests. The research question is clear, and the experimental design is easy to follow. In particular, the design in Table 1 separates what caregivers want from how they communicate it, allowing these factors to be examined separately.

Critical weaknesses

My main concerns are the validity of the benchmark and the scope of the study.

  1. The paper would benefit from a clearer account of how the benchmark was validated. Were the cases independently reviewed by clinical experts to confirm that the reference actions are appropriate and the alternatives are clinically inferior? It is also unclear how well the simulated caregivers reflect real caregiver behavior. The human study mainly evaluates model responses, so it does not fully address these questions.
  2. The study focuses on showing that models can give in to caregiver pressure, but offers less insight into how much the observed behavior is specific to caregiver-mediated settings. For example, it is unclear how these failures differ from those that arise when patients seek advice directly. The prompting experiment is a useful starting point, but the paper could say more about how the findings might guide improvements to these systems.
Questions for authors

Please see the weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZzS7

Thank you for your thoughtful review and for recognizing the value of separating caregiver preferences from communication style in our experiments. If these responses resolve your concerns, we would appreciate your reconsideration of the score.

Clinical validation and caregiver realism (Weakness 1)

Were the cases independently reviewed by clinical experts to confirm that the reference actions are appropriate and the alternatives are clinically inferior? It is also unclear how well the simulated caregivers reflect real caregiver behavior. The human study mainly evaluates model responses, so it does not fully address these questions.

We extended the original pediatric and eldercare conversation review to veterinary care, with a veterinarian (DVM) conducting the additional review. We also obtained clinical assessments of the benchmark cases themselves: all 32 reviewed cases across the three domains were judged well grounded in their cited clinical guidelines and professional reference materials. The veterinarian additionally judged 14/15 reference actions appropriate and 14/15 inferior actions clinically less appropriate. During the conversation reviews, reviewers described the caregiver follow-ups as sensible and realistic in qualitative feedback. Our shared response on clinical validation details these case and action assessments and the veterinary conversation review.

Patient–caregiver comparison and implications for alignment (Weakness 2)

For example, it is unclear how these failures differ from those that arise when patients seek advice directly. The prompting experiment is a useful starting point, but the paper could say more about how the findings might guide improvements to these systems.

Much of the work on mitigating LLM sycophancy focuses on preserving factually correct answers when users express conflicting views [1,2]. In multi-stakeholder settings, several plans may be appropriate, and choosing among them requires considering stakeholders’ roles, preferences and who bears the consequences [3,4]. Factual correctness alone does not resolve how an LLM should respond when these interests conflict. CareDistance examines this general alignment question in caregiver-mediated healthcare, evaluating clinical appropriateness separately from action agreement to distinguish acceptable accommodation from advice that compromises the patient’s care.

Following your suggestion, we ran the matched patient–caregiver comparison with the same clinical facts and inferior action, and found no consistent difference in susceptibility. In caregiver cases where patients could plausibly participate, LLMs often gave clinically inappropriate advice without seeking their wishes. We then tested whether explicit beneficiary input guides advice when preferences conflict. Keeping the caregiver’s inferior request unchanged, we varied whether the patient or another relative wanted the same recommended care. Stating the patient’s wishes reduced failures more than stating another relative’s preference, yet LLMs still gave clinically inappropriate advice in 17–41% of conversations across models despite explicitly knowing that the patient wanted recommended care.

These results identify an alignment failure: LLMs do not reliably account for the beneficiary’s role and interests when advising the user (caregiver), despite showing some sensitivity to whose preference is stated. First-party sycophancy evaluations alone cannot establish whether LLMs account for a separate beneficiary’s interests when that beneficiary’s wishes are unknown or conflict with the user’s preferences. These findings motivate alignment training and reward modeling that teach LLMs to identify affected beneficiaries, seek relevant input where feasible and warranted, and use it consistently when preferences conflict. Our shared response (Part 1/3, Part 2/3, Part 3/3) presents the controls and prompting experiments that inform these priorities for improving LLMs.

References

[1] Chen et al. From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning. ICML 2024.

[2] Li et al. Causally Motivated Sycophancy Mitigation for Large Language Models. ICLR 2025.

[3] Sorensen et al. Position: A Roadmap to Pluralistic Alignment. ICML 2024.

[4] Gabriel and Keeling. A matter of principle? AI alignment as the fair treatment of claims. Philosophical Studies, 2025.

Dear Reviewer ZzS7,

As the discussion period is close to coming to an end, we wanted to follow up and ask whether our responses and revised manuscript have addressed the concerns you raised. We would be happy to clarify or expand on any remaining points. We sincerely appreciate your time and thoughtful feedback, which have helped us strengthen the evaluation and clarify the paper’s contribution.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Shared response: User–beneficiary separation (Part 1/3)

We thank all three reviewers for their careful feedback. Reviewer ZzS7 highlights the separation of caregiver preferences from communication style. Reviewer tS6m calls CareDistance an “important problem with a useful benchmark” and values the neutral multi-turn control. Reviewer ytNU notes that these controls make preference effects “substantially more interpretable” and that separating clinical appropriateness from action agreement makes the evaluation clinically meaningful.

We have added the matched first-party comparison requested by the reviewers and find no consistent difference in susceptibility between patient and caregiver settings. CareDistance studies a general alignment question in multi-stakeholder interactions: whether LLMs account for affected stakeholders when the user’s preferences may conflict with the interests of a separate beneficiary who bears the consequences. We examine this in caregiver-mediated healthcare by measuring how sycophantic accommodation of the user can compromise the patient’s care. In cases where patients could plausibly participate, LLMs often gave clinically inappropriate advice without seeking their wishes. Stating that the patient wanted recommended care reduced failures more than attributing that preference to another relative, yet substantial inappropriate care persisted despite the patient’s wishes being explicitly known. Explicitly prompting the LLM to prioritize the patient also provided only partial protection. Together, these findings identify an alignment failure in multi-stakeholder interactions: LLMs do not reliably account for who bears the consequences of their advice and whose interests and preferences should guide it when the user and beneficiary differ.

1 The alignment requirement when the user and beneficiary differ

Prior work shows that human preference judgments and preference models can reward agreement with the user over truthful responses [1]. This also extends to social disagreements, where LLMs can agree with opposing sides depending on the user’s perspective, and preference datasets can reward this behavior [3]. Medical sycophancy under repeated user pushback has likewise been established [4]. Much of the work on mitigating LLM sycophancy focuses on preserving factually correct answers when users express conflicting views [2,8]. In multi-stakeholder care decisions, several plans may be clinically appropriate, and choosing among them requires considering stakeholders’ roles, preferences and who bears the consequences [5,6]. Factual correctness alone does not resolve how an LLM should respond when these interests conflict.

Pluralistic alignment recognizes the diversity of human preferences [5]. Within an interaction, models must also account for how stakeholders’ different roles and interests should influence their advice [6]. The person directing the conversation is not necessarily the beneficiary who bears its consequences. We strongly believe that First-party sycophancy evaluations alone cannot establish whether LLMs account for a separate beneficiary’s interests when that beneficiary’s wishes are unknown or conflict with the user’s preferences. Models should identify affected stakeholders, seek relevant input where feasible and warranted, and use available information appropriately throughout the conversation.

Caregiver-mediated healthcare makes this alignment question measurable: the caregiver directs the exchange with the LLM doctor, while the patient receives the care being recommended. Caregiver concerns can legitimately shape a workable plan, but do not themselves establish the patient’s wishes or justify clinically inappropriate care. Because more than one plan can be clinically appropriate [7], CareDistance assesses recommendations against the patient’s circumstances and clinical sources, separately from agreement with the reference action or the inferior action.

AuthorsOpenReview ↗

Shared response: User–beneficiary separation (Part 2/3)

2 Matched patient and caregiver conversations

Following the reviewers’ suggestion, we selected 112 eldercare and 37 pediatric cases where patients could plausibly ask the same care question themselves. We evaluated four models in conversations where either the patient or caregiver requested the same inferior action, keeping the clinical facts, model settings, adaptive follow-up protocol and evaluation procedures unchanged. For each model, we include only questions whose first neutral response was appropriate in both roles; the same set of questions are used throughout Tables 1 and 2. Table 1 reports the percentage of conversations with a reply judged clinically inappropriate by turn five, with 95% confidence intervals.

Table 1 Clinically inappropriate care by turn five

Model and nPatient NeutralPatient Inferior preferenceCaregiver NeutralCaregiver Inferior preference
Claude Sonnet 5 (n=119)11.8 [6.7–17.6]58.8 [49.6–67.2]13.4 [7.6–20.2]68.1 [59.7–76.5]
GPT-5.6 Sol (n=145)3.4 [0.7–6.9]31.0 [24.1–38.6]2.1 [0.0–4.8]36.6 [29.0–44.1]
GPT-5.6 Luna (n=141)2.8 [0.7–5.7]32.6 [25.5–40.4]2.1 [0.0–5.0]37.6 [29.8–45.4]
Gemini 3.8 Flash (n=134)13.4 [8.2–19.4]44.8 [36.6–53.0]16.4 [10.4–22.4]42.5 [34.3–51.5]

Inferior preferences produced substantially more failures than neutral follow-up in both roles, with no consistent difference in paired role comparisons. In the caregiver condition, patient wishes were initially unstated, and 37–68% of conversations contained clinically inappropriate advice. Among histories whose first clinically inappropriate reply also agreed with the inferior action, LLM-based annotation showed that 80–88% reached that reply without asking or suggesting checking what the patient wanted, even though patients could plausibly participate in these cases.

3 Known patient wishes and the source of the contrary preference

We next examined whether providing the patient’s wishes prevents inappropriate accommodation of the caregiver. The caregiver still requests the inferior action, but the opening message now also states that the patient wants recommended care. To test sensitivity to the beneficiary’s role, we ran another control in which a different relative wants that same recommended care. This comparison holds the added preference fixed while changing whose preference it is. The caregiver states the added preference once in the opening message and continues expressing the inferior preference in adaptive follow-ups. Clinical facts, model settings and evaluation procedures remain unchanged.

Table 2 Clinically inappropriate care by turn five with the caregiver still requesting inferior care

ModelPatient wishes initially unstatedPatient wants recommended careOther relative wants recommended care
Claude Sonnet 568.1 [59.7–76.5]41.2 [32.8–49.6]57.1 [47.9–65.5]
GPT-5.6 Sol36.6 [29.0–44.1]17.2 [11.7–23.4]33.1 [25.5–40.7]
GPT-5.6 Luna37.6 [29.8–45.4]24.1 [17.7–31.2]35.5 [27.7–43.3]
Gemini 3.8 Flash42.5 [34.3–51.5]24.6 [17.9–32.1]41.0 [32.8–49.3]

Percentages with a clinically inappropriate reply by turn five [95% CI]. Question-level bootstrap intervals are stratified by domain.

Even with the patient’s wishes explicitly stated, 17–41% of conversations still contained clinically inappropriate advice. In 17–31% of all eligible conversations, the first clinically inappropriate reply also agreed with the inferior action. Attributing the same recommended-care preference to the patient rather than another relative reduced clinical failures. Models therefore distinguish whose preference is stated, but that distinction does not reliably constrain the care they recommend when the caregiver prefers a conflicting course.

AuthorsOpenReview ↗

Shared response: User–beneficiary separation (Part 3/3)

4 Implications for alignment training and evaluation

The paper also includes a 150-case pediatric experiment testing an explicit instruction to prioritize appropriate patient care over inferior caregiver preferences (Appendix E.7). When the caregiver requested the inferior action, clinical-failure rates were 48.4% for Sonnet, 34.5% for Sol and 72.7% for Qwen, versus 64.8%, 37.3% and 74.7% without the instruction. Explicitly prioritizing appropriate patient care in the prompt therefore provided only partial protection and did not reliably prevent inappropriate caregiver accommodation.

Together, our experiments show that LLMs do not reliably maintain appropriate care for a separate beneficiary when the user prefers a conflicting course, despite some sensitivity to whose preference is stated. These failures motivate alignment training and reward modeling that account for stakeholders’ roles, preferences and who bears the consequences, including beneficiaries absent from the conversation. This requires identifying affected stakeholders, seeking relevant missing input where feasible and warranted, and using it consistently when preferences conflict. CareDistance makes this general alignment requirement measurable through controlled comparisons of stakeholder roles and preferences, alongside clinical evaluation of the resulting advice.

References

[1] Sharma et al. Towards Understanding Sycophancy in Language Models. ICLR 2024.

[2] Chen et al. From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning. ICML 2024.

[3] Cheng et al. ELEPHANT: Measuring and understanding social sycophancy in LLMs. ICLR 2026.

[4] Kim et al. The Doctor Will Agree With You Now: Sycophancy of Large Language Models in Multi-Turn Medical Conversations. 1st Workshop on Linguistic Analysis for Health (HeaLing 2026).

[5] Sorensen et al. Position: A Roadmap to Pluralistic Alignment. ICML 2024.

[6] Gabriel and Keeling. A matter of principle? AI alignment as the fair treatment of claims. Philosophical Studies, 2025.

[7] National Institute for Health and Care Excellence (NICE). Shared decision making. NICE guideline NG197, 2021.

[8] Li et al. Causally Motivated Sycophancy Mitigation for Large Language Models. ICLR 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.