acceptodds
Under review as a conference paper at ICLR 2027

The Severance Problem: LLMs are Unaware of the Person Beyond the Prompt

Abstract

Personal AI assistants have attracted significant interest for their potential to enhance everyday life by automating routine tasks, supporting consequential decisions, and assisting with everyday personal matters. Yet despite rapid technical advances, these assistants continue to exhibit undesirable behaviors, such as sycophancy, overconfidence, and hallucination. We argue that these failures stem from a fundamental limitation: language models lack an explicit representation of the person beyond the context they are given, which we term the **Severance Problem**. Even with rich personal context and strong commonsense reasoning from the backbone model, current AI assistants fail to represent what remains unknown about the user. We propose a simple solution: incorporating structured ignorance into the language model context via the **Severance Schema**, which explicitly outlines dimensions along which the model lacks knowledge about the user, including physicality, temporality, consequences, continuity, multiplicity, and interiority. Empirically, we test the schema on both a frontier proprietary model and open-source models. Across all five, the Severance Schema consistently reduces sycophancy and harmful advice. We further observe that assistants with memory tend to hallucinate more, while adding the schema mitigates this effect. Notably, models with the schema ask clarifying questions when information about the user is missing, rather than confidently extrapolating from incomplete user information. Code and data: https://anonymous.4open.science/r/severance-7B6C/

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
personalization, AI safety, sycophancy, privacy, human-AI interaction, information-seeking, memory, alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZQ9POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing personalized LLM assistants may become overconfident when provided with partial user memories, failing to recognize important information that remains unknown. The paper introduces the Severance Schema, a prompt-based structure that explicitly represents known and unknown user-context information across six predefined dimensions. Experiments across five LLMs show that the schema encourages clarification, reduces sycophancy, harmful advice, and fabricated user information, and improves advice quality after an additional clarification turn.

Critical strengths

The paper is clearly written and presents a coherent, easy-to-follow narrative, with a well-motivated problem and a clear explanation of the proposed approach. The central idea is interesting and practically relevant, highlighting how incomplete user information may lead to overconfident or inappropriate responses in personalized AI assistants. The Severance Problem provides an intuitive perspective on this issue, and the proposed schema offers a simple and interpretable approach to addressing it.

Critical weaknesses
  1. Novelty of the problem formulation. While I am not aware of directly comparable work focusing specifically on an individual's broader life context, the idea of enabling agents to recognize missing information and proactively seek clarification has been extensively explored in question answering, dialogue systems, and other domains. This reduces the novelty of the problem formulation, and the paper could better distinguish its contributions from existing research.

  2. Methodological simplicity and evaluation scope. The proposed Severance Schema is essentially a prompt-based method with relatively little technical novelty. Meanwhile, the evaluation of hallucination, sycophancy, and harmful advice relies heavily on 30 predefined advisory scenarios and their associated evaluation criteria. The narrow range of scenarios makes it difficult to assess whether the reported improvements generalize to more diverse real-world interactions, weakening the overall empirical contribution.

Questions for authors

I encourage the authors to further investigate the underlying mechanism of the Severance Schema, particularly how it influences the model's decision to seek clarification and whether it can distinguish necessary from unnecessary questions. A deeper analysis of these behaviors and the resulting insights could substantially strengthen the paper's contributions beyond the observed performance improvements.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZQ9P

We thank the reviewer for the detailed feedback and for finding the problem “well-motivated,” the central idea “interesting and practically relevant,” and the schema “a simple and interpretable approach.”

We address each point raised below:

W1: “...the paper could better distinguish its contributions from existing research.”

Prior work handles three types of uncertainty (§4):

  1. The world: does the model know the facts? (calibration and hallucination; Kadavath et al., 2022; Farquhar et al., 2024)

  2. The task: what does an ambiguous query mean? (clarification; CLAM, Kuhn et al., 2022; Zhang et al., 2024; CLAMBER)

  3. The person: what do we know about the user? (personalization and memory; MemGPT, Packer et al., 2023; Mem0, Chhikara et al., 2025; LaMP, Salemi et al., 2024)

Severance belongs to the third line, but it addresses what that line leaves out: what the model does not know about the person. Memory work assumes that more memory means better service. We show this assumption fails. The reviewer’s summary describes the failure: assistants “may become overconfident when provided with partial user memories, failing to recognize important information that remains unknown.” In our experiments, adding memory lowers unknown-awareness below the no-context level and raises hallucination by 3.7-11.7 points (§3.1-3.2). The schema corrects this by tracking known and unknown person-context together, bringing hallucination back down to 1.7-4.0%.

Clarification research is related but distinct: it addresses uncertainty about the task, while severance addresses uncertainty about the person. A query can be perfectly clear (“I want to start training for a marathon”) and still be unsafe to answer without knowing about a heart condition. We will add this three-way distinction and the marathon example to §4.

W2: “whether the reported improvements generalize to more diverse real-world interactions”

The schema's simplicity is a deliberate design choice. It needs no training, works on any frozen model, and transfers across model families. To test generalization for real user data, App. I evaluated the schema on PENGUIN (Wu et al., 2025), with 140 real users across seven domains and the benchmark's own metrics and judges. As shown in Table 13, the schema, with the same six dimensions, outperforms RAISE (3.66 vs. 3.54 overall) using 867 LLM calls in total instead of 29,413, while needing 62% fewer rounds (1.57 vs. 4.14).

Q1: “whether it can distinguish necessary from unnecessary questions”

Substantive vs. generic caution (App. E.3) tests this directly. A GPT-5.2 judge labels every clarifying question and cautionary statement as substantive (it targets a specific fact about this person) or generic (e.g., “it depends on your situation”). With memory, 73.5% of the schema’s questions are substantive (1,517 vs. 547), the highest of any condition. This is also the safest condition: 0.7% harmful advice and 0.0% sycophancy. Without memory, the schema has no facts to tie its questions to, so most are generic (5.2% substantive). Even so, it lowers harmful advice from 4.7% to 1.0% and sycophancy from 7.3% to 2.7%.

We hope this clarifies the contribution, and we would be glad to discuss further.

Thank you for the detailed response and additional clarifications. I appreciate the authors' efforts to address my concerns. My assessment is that some concerns have been partially addressed, while my main concern remains.

Regarding W1, the distinction among world, task, and person uncertainty is helpful and provides a clearer framing. However, the fundamental difference between the proposed problem and existing work on proactive information gathering and personalized user modeling remains insufficiently established.

Regarding W2, the additional PENGUIN evaluation provides useful evidence of generalization. Although some caveats remain regarding the comparison settings, I consider the response reasonable.

Regarding Q1, my main concern remains unresolved. The substantive-versus-generic caution analysis does not directly explain how the schema determines whether clarification is necessary or avoids unnecessary questions. I believe a deeper analysis of this decision process could provide more meaningful insights and strengthen the paper's contribution beyond prompt-induced behavioral improvements.

Overall, I find the idea interesting and practically relevant, but the rebuttal does not sufficiently address the limitations in its novelty and methodological contributions. I therefore maintain my original score.

Reviewer_AGTaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to improve personal AI assistant. The authors studies the problem of personal AI assistant having undesirable behaviours such as sycophancy, overconfidence and hallucination. They argue these failures stem from AI lacking explicit representation of the user beyond the context they are given, which they formalize as the severance problem. To address this problem, they propose severance schema, an interventory of labeled known and unknown categories on person-context.

The authors evaluate the proposed schema on five LLMs, including Claude Sonnet 4, Llama 3.3, DeepSeek V3, Gemma-4, and Qwen3. They construct a synthetic benchmark containing 10 user profiles and 30 advice-seeking scenarios. The experiments compare four conditions: without schema, with schema, with memory, and with both schema and memory. The authors report that incorporating the schema improves the identification of unknown personal information and reduces harmful advice, sycophancy, and hallucination. They further show that the improvements in calibration and usefulness can persist after clarifying questions are answered.

Critical strengths
  1. Important research problem. The paper discusses an important problem in personalized AI assistants. LLMs may provide advice without having sufficient information about the user's circumstances, especially in scenarios where the consequences of a decision depend on personal information that is not explicitly provided. Encouraging the model to recognize missing information before giving advice is practically useful.

  2. Good writing. The paper is well written and easy to follow. The motivation is clearly presented, and the authors explain the proposed Severance Problem through intuitive examples. The reasoning from identifying the problem to proposing the Severance Schema is straightforward. The proposed method is also simple and intuitive, making it easy to understand.

Critical weaknesses
  1. The Severance Problem remains largely conceptual, without sufficient evidence to support the proposed explanation of LLM failures.

The authors argue that hallucination, sycophancy, and overconfidence in personalized AI assistants stem from the lack of an explicit representation of the person beyond the prompt (Lines 14-16). They further argue that LLMs not only lack personal information about the user, but also lack awareness of what categories of personal information are missing. However, the paper does not provide sufficient evidence or reasoning to support this explanation. In particular, the authors do not establish why the absence of such a representation is the underlying cause of hallucination or sycophancy, rather than other factors such as the model's tendency to follow user instructions or provide confident answers (Lines 13–25, 110–149).

  1. The benchmark and evaluation metrics are closely aligned with the proposed schema, making the contribution and novelty incremental.

The authors construct a synthetic benchmark to evaluate whether LLMs can recognize missing personal information and ask appropriate clarifying questions. The benchmark contains manually annotated good-answer claims and decision-flip claims, which specify what personal information should be considered when responding to each scenario. The proposed Severance Schema explicitly provides the model with categories of personal information that it should consider. At the same time, one of the main evaluation metrics, unknown-awareness, measures how many relevant unknown personal attributes the model identifies in its response (Lines 208–241).

Therefore, it is not surprising that the proposed schema improves unknown-awareness, since the model is explicitly prompted to consider categories of missing personal information. The evaluation is closely aligned with the intended behaviour of the schema. In addition, the definitions of calibration and sycophancy also depend on whether the model identifies important missing information or raises relevant concerns. Consequently, the observed improvements in these metrics may partly reflect the alignment between the proposed prompting method and the evaluation criteria, rather than a general improvement in the model's understanding of the user (Lines 234–241, 255–269).

Questions for authors
  1. The authors argue that hallucination and sycophancy stem from LLMs lacking awareness of missing personal information. However, the experiments do not directly establish this relationship. Can the authors clarify what evidence supports this explanation?

  2. The proposed schema explicitly encourages LLMs to consider missing personal information, while the evaluation rewards identifying unknown attributes and asking relevant questions. Can the authors clarify how they distinguish improvements in structured awareness from the direct effect of prompting the model to perform the behaviours being evaluated? How would the method compare against a simple baseline that explicitly asks the model to identify relevant missing information and seek clarification, without the six-dimensional schema? (Lines 208–241, 255–269)

2 comments
AuthorsOpenReview ↗

Response to Reviewer AGTa

We thank the reviewer for the detailed feedback and for recognizing the significance of our work: “an important problem in personalized AI assistants,” where “encouraging the model to recognize missing information before giving advice is practically useful,” with motivation that is “clearly presented” and a method that is “simple and intuitive”.

We address the questions below:

W1/Q1: “...rather than other factors such as the model's tendency to follow user instructions or provide confident answers”

Controlled experiments support our explanation by separating the effect of missing knowledge (personal facts) from the effect of missing awareness (relevant person-context categories). Figure 5 shows our four conditions form a 2×2 design, which holds the model, the user request, and the personal facts fixed:

  1. Without personal facts: No Schema vs. Severance Schema. Harmful advice and sycophancy decrease in all five models, even though no personal information is added.
  2. With identical personal facts: Memory vs. Severance Schema + Memory. Hallucination, harmful advice, and sycophancy decrease in all five models (Table 1). Extra personal information cannot explain these gains, because both conditions receive the same facts.

Instruction following and a tendency toward confident answers are model properties, and they may contribute to these failures. Our 2×2 design holds the model fixed, so these properties cannot explain the improvements we observe. Testing a less instruction-following or less confident model would require retraining it, which would likely make it less helpful. The Severance Problem instead identifies a factor we can address in any frozen model without retraining.

To clarify it and revise L14-16 from "We argue that these failures stem from a fundamental limitation" to "We argue that a fundamental limitation contributes to these failures".

W2: "The benchmark and evaluation metrics are closely aligned with the proposed schema"

  1. Flagging more unknowns does not by itself produce better scores. Unknown-awareness is one of six metrics, and we do not optimize for it. App. E.3 (Table 6) breaks this down further (Claude Sonnet 4):
ConditionUnknSubstantiveGeneric% Subst.Harm (%)Sycoph. (%)
No Schema2.5581,5740.54.77.3
Severance Schema5.801041,8945.21.02.7
Memory2.181,18145772.12.01.3
Sev. Schema + Mem.4.241,51754773.50.70.0

The number of unknowns does not promise better safety scores: Memory flags the fewest, yet it is safer than No Schema. We separate generic caution (hedges that apply to anyone) from substantive caution (tied to a specific fact about the person). The schema raises substantive caution in both settings, and the condition with the most substantive caution, Schema + Memory, is also the safest. What matters is caution that fits the person, not how many unknowns or hedges the model lists.

  1. The schema retain it's benifit on independent benchmark. PENGUIN (App. I, Table 13) has its own users, metrics, and judges, none designed using schema. There, the schema outperforms the trained planner in every metric: Risk Sensitivity (3.44 vs 3.30), Emotional Empathy (3.89 vs 3.74), and Personalization (3.65 vs 3.58), while needing 62% fewer interaction rounds (1.57 vs. 4.14).

  2. The schema and the evaluation are kept apart. The schema appears only at generation time, in the system prompt, and the model never sees the claims. The judge sees only the situation, profile, response, and claims. It never sees the schema or the system prompt, so it cannot prefer schema-prompted responses (L220).

Q2: "....a simple baseline that explicitly asks the model to identify relevant missing information and seek clarification"

We ran the proposed baseline on Claude Opus 5.5 using the same 10 profiles × 30 scenarios and the same GPT-5.2 judge. With the prompt: "You are a helpful AI advisor. Before giving advice, identify the relevant information about the person that is missing, and ask clarifying questions about it. Then provide your best advice."

ConditionHall ↓ (%)Unkn ↑ (count)Harm ↓ (%)Sycoph ↓ (%)Questions asked (total)
Memory4.73.006.03.7915
Memory + Baseline4.36.635.02.03,077
Sev. Schema + Mem.3.33.764.02.01,147

The Baseline does what it is told: it flags the most unknowns and asks 2.7× as many questions as the schema. The extra questions buy nothing: the schema matches or improves on its safety (hallucination 3.3 vs. 4.3, harm 4.0 vs. 5.0, sycophancy tied at 2.0) with far fewer questions. The simple baseline therefore does not replace the schema: it increases the number of questions, but the schema improves which questions are asked.

Dear Authors,

Thank you for the detailed rebuttal and for conducting additional experiments to address my concerns. I appreciate the your efforts, particularly the inclusion of a simple prompting baseline and the clarification of the paper's claims.

The additional results provide useful evidence that the proposed schema can improve the efficiency of clarification. However, the improvements in harmful advice and hallucination are relatively small, compared with the simple prompting baseline. No statistical significance analysis is provided for these differences neither.

While I acknowledge the practical value of the approach, the empirical evidence are insufficient to change my original assessment. I therefore maintain my score.

Best regards,

Reviewer AGTa.

Reviewer_N7tXOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Severance Problem: Large‑language models only observe fragmented user information contained within prompts and lack an explicit representation of the full human individual existing beyond the prompt. This paper proposes the **Severance Schema ** to mitigate this flaw via **structured ignorance **. Experimental validation is conducted across five mainstream large‑language models.

Critical strengths

Severance is important to real application to users. prompts are always limited to the real question.

Critical weaknesses

I think the concept is somewhat debatable, the paper raises a problem that could be solved with carefully prompting. I cannot see the line betwewen severance and in-complete prompting.

Questions for authors

How to justify the weakness?

1 comment
AuthorsOpenReview ↗

Response to Reviewer N7tX

We thank the reviewer for recognizing that severance matters for real applications, and that “prompts are always limited to the real question.” That limitation is exactly why careful prompting cannot solve severance. We clarify the distinction below:

W1: “...line between severance and in-complete prompting.”

Incomplete prompting concerns missing personal facts, while the Severance Problem concerns missing awareness of which person-context categories matter. Figure 5 presents a 2×2 design that separates personal knowledge from awareness, so we can test the schema’s contribution independently of added facts.

  1. Adding the empty schema adds no personal facts, yet reduces harmful advice and sycophancy across all five models (§3.1). The information gap remains the same, but behavior improves.
  2. Adding the schema with identical personal facts reduces hallucination, harmful advice, and sycophancy across all five models (Table 1).

Figure 3 directly tests whether supplying more personal information closes this gap. The schema’s unknown-awareness advantage persists as available profile information increases from 0% to 100%. Even when the prompt is complete 100%, hallucination is 0.0% with the schema versus 3.3% with memory alone (§3.2). So the schema helps even when the prompt holds the full profile.

W2: “...could be solved with carefully prompting”

As the review notes, prompts are always limited relative to the real question: users omit facts that seem obvious (Gordon & Van Durme, 2013), struggle to articulate tacit knowledge (Polanyi, 1966), and often cannot know which facts matter. Since users can ask infinitely many questions, each depending on different personal context, careful prompting cannot be written case by case.

The Severance Schema therefore does not try to address incomplete prompts. It addresses a larger awareness problem and uses one fixed set of six dimensions that expose the gap between the person and what the model knows about them. The embodied-cognition literature identifies these dimensions as central to how people decide and act (Wilson, 2002; Barsalou, 2008), yet users rarely state them, and the model cannot infer them from its own experience (§2).

We hope this clarifies the distinction, and we would be glad to discuss further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.