- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Existing personalized LLM assistants may become overconfident when provided with partial user memories, failing to recognize important information that remains unknown. The paper introduces the Severance Schema, a prompt-based structure that explicitly represents known and unknown user-context information across six predefined dimensions. Experiments across five LLMs show that the schema encourages clarification, reduces sycophancy, harmful advice, and fabricated user information, and improves advice quality after an additional clarification turn.
Critical strengths
The paper is clearly written and presents a coherent, easy-to-follow narrative, with a well-motivated problem and a clear explanation of the proposed approach. The central idea is interesting and practically relevant, highlighting how incomplete user information may lead to overconfident or inappropriate responses in personalized AI assistants. The Severance Problem provides an intuitive perspective on this issue, and the proposed schema offers a simple and interpretable approach to addressing it.
Critical weaknesses
-
Novelty of the problem formulation. While I am not aware of directly comparable work focusing specifically on an individual's broader life context, the idea of enabling agents to recognize missing information and proactively seek clarification has been extensively explored in question answering, dialogue systems, and other domains. This reduces the novelty of the problem formulation, and the paper could better distinguish its contributions from existing research.
-
Methodological simplicity and evaluation scope. The proposed Severance Schema is essentially a prompt-based method with relatively little technical novelty. Meanwhile, the evaluation of hallucination, sycophancy, and harmful advice relies heavily on 30 predefined advisory scenarios and their associated evaluation criteria. The narrow range of scenarios makes it difficult to assess whether the reported improvements generalize to more diverse real-world interactions, weakening the overall empirical contribution.
Questions for authors
I encourage the authors to further investigate the underlying mechanism of the Severance Schema, particularly how it influences the model's decision to seek clarification and whether it can distinguish necessary from unnecessary questions. A deeper analysis of these behaviors and the resulting insights could substantially strengthen the paper's contributions beyond the observed performance improvements.
2 comments
Thank you for the detailed response and additional clarifications. I appreciate the authors' efforts to address my concerns. My assessment is that some concerns have been partially addressed, while my main concern remains.
Regarding W1, the distinction among world, task, and person uncertainty is helpful and provides a clearer framing. However, the fundamental difference between the proposed problem and existing work on proactive information gathering and personalized user modeling remains insufficiently established.
Regarding W2, the additional PENGUIN evaluation provides useful evidence of generalization. Although some caveats remain regarding the comparison settings, I consider the response reasonable.
Regarding Q1, my main concern remains unresolved. The substantive-versus-generic caution analysis does not directly explain how the schema determines whether clarification is necessary or avoids unnecessary questions. I believe a deeper analysis of this decision process could provide more meaningful insights and strengthen the paper's contribution beyond prompt-induced behavioral improvements.
Overall, I find the idea interesting and practically relevant, but the rebuttal does not sufficiently address the limitations in its novelty and methodological contributions. I therefore maintain my original score.
Response to Reviewer ZQ9P
We thank the reviewer for the detailed feedback and for finding the problem “well-motivated,” the central idea “interesting and practically relevant,” and the schema “a simple and interpretable approach.”
We address each point raised below:
Prior work handles three types of uncertainty (§4):
The world: does the model know the facts? (calibration and hallucination; Kadavath et al., 2022; Farquhar et al., 2024)
The task: what does an ambiguous query mean? (clarification; CLAM, Kuhn et al., 2022; Zhang et al., 2024; CLAMBER)
The person: what do we know about the user? (personalization and memory; MemGPT, Packer et al., 2023; Mem0, Chhikara et al., 2025; LaMP, Salemi et al., 2024)
Severance belongs to the third line, but it addresses what that line leaves out: what the model does not know about the person. Memory work assumes that more memory means better service. We show this assumption fails. The reviewer’s summary describes the failure: assistants “may become overconfident when provided with partial user memories, failing to recognize important information that remains unknown.” In our experiments, adding memory lowers unknown-awareness below the no-context level and raises hallucination by 3.7-11.7 points (§3.1-3.2). The schema corrects this by tracking known and unknown person-context together, bringing hallucination back down to 1.7-4.0%.
Clarification research is related but distinct: it addresses uncertainty about the task, while severance addresses uncertainty about the person. A query can be perfectly clear (“I want to start training for a marathon”) and still be unsafe to answer without knowing about a heart condition. We will add this three-way distinction and the marathon example to §4.
The schema's simplicity is a deliberate design choice. It needs no training, works on any frozen model, and transfers across model families. To test generalization for real user data, App. I evaluated the schema on PENGUIN (Wu et al., 2025), with 140 real users across seven domains and the benchmark's own metrics and judges. As shown in Table 13, the schema, with the same six dimensions, outperforms RAISE (3.66 vs. 3.54 overall) using 867 LLM calls in total instead of 29,413, while needing 62% fewer rounds (1.57 vs. 4.14).
Substantive vs. generic caution (App. E.3) tests this directly. A GPT-5.2 judge labels every clarifying question and cautionary statement as substantive (it targets a specific fact about this person) or generic (e.g., “it depends on your situation”). With memory, 73.5% of the schema’s questions are substantive (1,517 vs. 547), the highest of any condition. This is also the safest condition: 0.7% harmful advice and 0.0% sycophancy. Without memory, the schema has no facts to tie its questions to, so most are generic (5.2% substantive). Even so, it lowers harmful advice from 4.7% to 1.0% and sycophancy from 7.3% to 2.7%.
We hope this clarifies the contribution, and we would be glad to discuss further.