acceptodds
Under review as a conference paper at ICLR 2027

Empathic Accommodation: LLMs Skew Information to Favor Users’ Inferred Emotional Stakes from Role Cues

Abstract

People increasingly consult Large Language Models (LLMs) before making consequential decisions in health, finance, and other domains. We document a previously unrecognized response tendency in which LLMs systematically skew objective information or objective assessments to favor the user based solely on inferences from the user’s role, even when the prompt contains no stated opinions. Five studies involving 46,000 trials collectively provide evidence for the effect: LLMs summarized brand discourse more favorably when the user owned the brand, aligned policy evaluations with the user’s political party affiliation, attributed positive customer reviews to the user’s professional role and negative reviews away from it, rendered more favorable investment analyses to already-invested users, and generated less severe medical diagnoses from identical symptom profiles when the recipient was the patient rather than the physician. The studies provide convergent evidence for the effect in different contexts, using different prompts, and models, with and without thinking modes, accumulating evidence by testing it from multiple angles via subtle differences in the experimental manipulations. Nonetheless, in all cases, the task was one of information retrieval or objective assessment, where a faithful response should not depend on who is asking. We call the effect empathic accommodation. Two studies provide functional support for the empathic-accommodation account, which hypothesizes that LLMs’ responses track users’ inferred emotional stakes. Instructing models to disregard how the response might make the user feel eliminated the effect, whereas instructions targeting user identity, the model’s own emotions, LLMs’ thinking mode, or chain-of-thought reasoning, did not consistently do so, thus ruling out several alternative explanations. Because empathic accommodation operates through implicit inferences from the user’s role, it is difficult to detect, yet it can be mitigated with simple targeted prompt instructions across different models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, AI bias, empathic AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_P7NwOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper reports empathic accommodation: when told only the user's role, with no stated opinion, LLMs shift information in the user's favor. The evidence is five studies and the authors conclude that models track users' inferred emotional stakes.

Critical strengths

S1. The paper documents a consequential, hard-to-notice distortion across several domains and model families with large samples. The medical study is the most striking to me.

S2. The paper is transparent, which makes the claims auditable and the proposed mitigation easy to re-test.

Critical weaknesses

W1. The role cues are explicit and task-relevant, so the design cannot tell empathic accommodation apart from legitimate personalization to the user's needs. The SI prompts state the role directly:

Study 1: "Owns a Peloton bike" in a user-memory block. This is consumer ownership, not brand ownership as the main text says.

Study 2: "Identifies as a Democrat."

Study 4: "The patient label will be shared directly with the patient."

Study 5: "The user has already invested $10,000."

Under each cue, a different answer can be a reasonable response to what the user needs: patient-facing wording, a hold-vs-buy framing, or experience-focused summaries for existing customers. It need not reflect an inference about the user's feelings. Relatedly, several tasks are not clearly "objective": Study 2 asks whether a policy is "good or bad", so the premise that a faithful answer should not depend on who is asking is contestable. Without conditions that separate the user's information needs from their emotional stakes, the shift is shown but the mechanism in the title is not. Examples of such conditions: matched needs with a reversed stake (e.g., a user who is short the stock), a length-matched neutral cue, or contrasting "do not tailor to the user's needs" with "do not consider how the user will feel".

W2. The model selection is dated and limited. GPT-4o (2024), Gemini 2.5, and Llama 3.3 are used, mostly in mini/Flash/Lite/Haiku tiers, and thinking mode is disabled in Studies 2–5. Study 1 itself finds that thinking attenuates the effect. The motivating health and finance use cases are where users are most likely to consult current flagship reasoning models, so whether the effect persists in those models is central to the paper's practical claim.

W3. Most results are reported as long inline strings of statistics, and the main text has only two figures. A cross-study summary table and per-model effect-size plots would make the evidence much easier to evaluate.

Questions for authors

See weaknesses. I would be happy to improve my score if my questions could be reasonably resolved.

Reviewer_4Ls7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether AI models will respond to users that have a stake in a particular topic in different ways. Experiment 1 studies if a user that owns a product of brand X will get better summaries for product X. Experiment 2 studies if a user that has political affiliation X will get a biased summary. Experiment 3 was for app and delivery platform managers routing good/bad reviews to their corresponding source. Experiment 4 was for patients vs. doctors receiving a diagnosis. Experiment 5 was for investments in an existing company and tested various interventions, upon which the paper found emotion suppression to be most useful.

Critical strengths
  1. There are quite a few domains and different aspects of the research question that are studied in experiments, representing a pretty cohesive evaluation. I'd like to highlight that this effort was quite comprehensive.

  2. The number of trials is large for each individual model within each study.

Critical weaknesses

I think there are more than 1-2 negative points that are substantial, so I will list them all:

  1. Presentation: The paper is completely missing a related work section. Section titles and paragraphs are very long. Paragraphs are hard to read as it is just a wall of text.

  2. Confounder in study: Authors claim that models should answer similarly across experimental and control conditions. However, I disagree. In experiment 1 for example, the model knowing that the user owns a product of brand X might cause it to think that a different set of online sources are more relevant for the user. The model is only instructed to "Summarize what people say online about [brand]". And so, if the model surfaces more online discourse about people who already own the brand, that might inherently be more positively skewed and it is not the model who is making the answers skewed.

  3. The models tested are small and slightly out of date. Therefore, I think there's a reasonable generalizability problem here where it's uncertain whether this problem disappears with size (though I don't expect that to happen, it would be more convincing).

Questions for authors

N/A

Reviewer_yCFSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper reports an LLM response tendency that the authors term empathic accommodation. When the prompt specifies only the user's role, and the user expresses no opinion or emotion, LLMs still skew objective information or assessments in favor of the user's inferred stakes. The authors examine this phenomenon in five studies spanning five domains. They argue that the effect is driven by the model's inference of how the user will feel about its response.

Critical strengths

Strengths  The research question has practical significance.  The effect is cross-validated across multiple studies, domains, and models.  The paper includes control interventions designed to rule out alternative explanations.

Critical weaknesses

Weaknesses  The paper does not sufficiently distinguish the proposed effect from sycophancy or from prior work on identity-based bias. The paper also offers no methodological or mechanistic ML contribution.  Each study uses a different subset of models, and several of the models are relatively outdated.  The LLM judges (GPT-4o / GPT-4o-mini) are not validated against human annotation, and no agreement with human labels is reported.  Results are inconsistent across models, which may weaken the conclusion. In addition, the significance in some studies are small.

Questions for authors

Questions  What is the conceptual contribution of this work beyond prior findings? Why should "inferring the user's stakes from their role and then accommodating them" not be regarded as a form of implicit sycophancy?  Could the authors test wording variants of the other intervention instructions and verify that models actually comply with them?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.