- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CareDistance, a benchmark for studying third-party sycophancy in caregiver-mediated healthcare, where the person interacting with an AI system is distinct from the patient who bears the consequences of its advice. The benchmark contains 450 source-grounded cases across eldercare, pediatrics, and veterinary care. For each fixed clinical case, the authors vary caregiver preferences and five communication personas, yielding 13 experimental conditions, and evaluate advisor behavior over conversations of up to five turns.
Critical strengths
- The paper is well organized, and the experimental logic is easy to reconstruct.
- The paper does not simply compare a neutral question with an adversarial prompt. Patient facts are kept fixed, caregiver preference and communication style are factorially varied, neutral follow-up provides a multi-turn control, and a gold-preference condition provides an additional sanity check. The authors also explicitly prevent the caregiver simulator from introducing new clinical facts or objective constraints during follow-up. This is consequential because it makes the observed preference-associated differences substantially more interpretable than an ordinary adversarial-prompt benchmark.
- CareDistance spans three beneficiary-dependent healthcare settings, five decision categories, 450 clinical cases, multiple model families, and conversations extending to five turns. The resulting effect is not confined to one model: inferior preference increases switching beyond neutral follow-up across all six models evaluated in all three domains, with large effect sizes for several systems. The source-grounded formulation and explicit separation of action agreement, overall care appropriateness, and potential harm also make the benchmark more clinically meaningful than purely stance-based sycophancy metrics.
Critical weaknesses
-
The paper convincingly demonstrates that caregiver pressure can shift models toward clinically inferior recommendations. However, it does not include the natural counterfactual in which the same clinical facts and same inferior preference are expressed by the patient directly rather than by a caregiver. Therefore, the experiments establish sycophancy in a user–beneficiary-separated setting, but do not establish whether user–beneficiary separation changes susceptibility relative to ordinary first-party medical sycophancy. This matters because closely related multi-turn medical sycophancy has already been demonstrated in prior work. A relatively small matched first-party experiment on the human-care domains could substantially sharpen the novelty claim. Alternatively, the authors should frame user–beneficiary separation as an important application/measurement setting rather than implying that the experiments identify a separation-specific mechanism.
-
The 450 cases, reference actions, and inferior alternatives are generated and verified primarily through LLM-based procedures grounded in clinical sources. The main experimental labels are then produced by one GPT-6 Luna judge. Human validation is restricted to 60 conversations from 30 automatically stratified pairs, two medical trainees, and no veterinary cases; additionally, the human estimates indicate substantially more neutral degradation than the automatic judge detects. This does not invalidate the qualitative finding, but the current evidence is weaker for precise claims about the magnitude of switching and especially the reported distribution of moderate/severe clinical harm. Larger independent expert validation—including veterinarians and validation of case/reference labels and harm severity—would materially strengthen the paper.
Questions for authors
-
Can the authors test a matched control where the patient is the user, at least for a representative small set of eldercare and pediatric cases? In this control, the patient states the same worse preference. All other clinical facts are the same. If the result is similar to the caregiver condition, I would see the main contribution mainly as a clinical extension of existing medical sycophancy work. If separation clearly changes behavior, this would greatly strengthen the main claim.
-
How much do the conclusions depend on the choice of automatic judge and the information it gets? The judge is given the separate neutral first response, the conversation being evaluated, and the source evidence.
3 comments
Response to Reviewer ytNU (Part 2/2)
Clinical validation and potential harm (Weakness 2)
Larger independent expert validation—including veterinarians and validation of case/reference labels and harm severity—would materially strengthen the paper.
We extended the original pediatric and eldercare conversation review to veterinary care and added clinical assessments of the benchmark cases. All 32 reviewed cases were judged well grounded in their cited clinical sources. A veterinarian (DVM) also judged 14/15 reference actions appropriate and 14/15 inferior actions clinically less appropriate.
For veterinary response validation, the veterinarian reviewed 30 five-turn conversations with model identities and previous judgments hidden. Agreement with GPT-6 Luna was 96.7% (κ = 0.934) on the first clinically inappropriate turn or on care remaining appropriate throughout.
To examine the effect-size discrepancy, an additional medical trainee adjudicated 19 disputed conversations and five agreement controls. The discrepancy mainly concerned additional failures identified by human reviewers in neutral conversations. Incorporating the adjudicated decisions gives an excess of 30.0 percentage points under inferior preference compared with neutral follow-up (95% CI: 16.7–43.3), versus 46.7 points under the original LLM judge. The new veterinary review found a corresponding excess of 33.3 points (95% CI: 20.0–46.7).
The additional case assessments and human–LLM agreement results across all three domains, together with adjudication of disputed judgments, strengthen confidence in the benchmark’s clinical grounding and evaluation, and support a substantial increase in inappropriate care under inferior caregiver preferences. We agree that precise effect sizes and the distribution of harm severity require broader clinical validation. Our shared response on clinical validation provides the full assessments and comparisons.
Sensitivity to the judge and neutral reference (Q2)
How sensitive are the conclusions to the choice and information available to the automatic judge? The judge is explicitly given the separate neutral first response in addition to the evaluated conversation and source evidence.
We tested whether the increase in inappropriate care under inferior preference depends on the judge or its access to the separate neutral reference. We used saved neutral and inferior-preference conversations for Claude Sonnet 5, GPT-5.6 Sol, GPT-5.6 Luna and Gemini 3.8 Flash, with 120 questions per advisor (40 each in eldercare, pediatrics and veterinary care).
Using the same conversations, clinical evidence and rubric, we compared:
-
Repeated judging: GPT-6 Luna with the original full input.
-
Without neutral reference: GPT-6 Luna, removing only the separate neutral first exchange.
-
Different judge: Claude Sonnet 5 with the original full input.
Increase in inappropriate care under inferior preference
| Advisor model | Original GPT-6 Luna | Repeat GPT-6 Luna | GPT-6 Luna without neutral reference | Claude Sonnet 5 (full original input) |
|---|---|---|---|---|
| Claude Sonnet 5 | 53.3 [43.1–63.1] | 51.7 [41.0–62.1] | 51.7 [41.2–61.7] | 59.0 [47.6–68.5] |
| GPT-5.6 Sol | 29.2 [20.6–38.1] | 25.0 [16.9–33.7] | 32.5 [24.1–40.9] | 34.2 [25.0–43.3] |
| GPT-5.6 Luna | 28.3 [19.7–37.1] | 33.3 [24.8–42.1] | 30.0 [21.6–38.7] | 31.7 [21.1–42.5] |
| Gemini 3.8 Flash | 26.7 [16.8–36.4] | 25.8 [14.7–36.6] | 21.7 [10.5–32.8] | 35.0 [25.4–44.7] |
Inferior preference minus neutral follow-up, in percentage points (95% CIs).
The increase in inappropriate care under inferior preference remains consistent across judges and settings, with only modest differences in estimated magnitude.
Dear Reviewer ytNU,
As the discussion period is close to coming to an end, we wanted to follow up and ask whether our responses and revised manuscript have addressed the concerns you raised. We would be happy to clarify or expand on any remaining points. We sincerely appreciate your time and thoughtful feedback, which have helped us strengthen the evaluation and clarify the paper’s contribution.
Response to Reviewer ytNU (Part 1/2)
Thank you for your thoughtful review and for recognizing the value of our experimental controls and the distinction between clinical appropriateness and action agreement. If these responses resolve your concerns, we would appreciate your reconsideration of the score.
Matched patient–caregiver comparison (Weakness 1 and Q1)
Following your suggestion, we ran the matched patient–caregiver comparison with the same clinical facts and inferior action, and found no consistent difference in susceptibility. CareDistance studies a general alignment question in multi-stakeholder interactions: whether LLMs account for a separate beneficiary’s interests when the user prefers a conflicting course. In caregiver-mediated healthcare, several plans may be clinically appropriate, and choosing among them requires considering stakeholders’ roles, preferences and who bears the consequences [1,2]. Preserving factually correct answers, the focus of much work on mitigating LLM sycophancy [3,4], does not by itself resolve how an LLM should respond when these interests conflict. We evaluate clinical appropriateness separately from action agreement to distinguish acceptable accommodation from advice that compromises the beneficiary’s care.
In selected caregiver cases where patients could plausibly participate, LLMs often gave inappropriate advice without seeking their wishes. We then tested whether explicit beneficiary input guides advice when preferences conflict. Keeping the caregiver’s inferior request unchanged, we varied whether the patient or another relative wanted the same recommended care. Stating the patient’s wishes reduced failures more than stating another relative’s preference, yet LLMs still gave clinically inappropriate advice in 17–41% of conversations across models despite explicitly knowing that the patient wanted recommended care.
These results identify an alignment failure: LLMs do not reliably account for the beneficiary’s role and interests when advising the user (caregiver), despite showing some sensitivity to whose preference is stated. First-party sycophancy evaluations alone cannot establish whether LLMs account for a separate beneficiary’s interests when that beneficiary’s wishes are unknown or conflict with the user’s preferences. Our controlled comparisons of stakeholder roles and preferences, alongside clinical evaluation of the resulting advice, provide evidence to guide alignment training and reward modeling for multi-stakeholder interactions. The shared response (Part 1/3, Part 2/3, Part 3/3) presents these experiments, results and alignment implications.
References
[1] Sorensen et al. Position: A Roadmap to Pluralistic Alignment. ICML 2024.
[2] Gabriel and Keeling. A matter of principle? AI alignment as the fair treatment of claims. Philosophical Studies, 2025.
[3] Chen et al. From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning. ICML 2024.
[4] Li et al. Causally Motivated Sycophancy Mitigation for Large Language Models. ICLR 2025.