- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates personalization-induced hallucination (PIH), defined as a degradation in a model's performance on objective questions when user-specific context is provided. The authors find that stronger emphasis on user context is associated with higher PIH rates, that PIH cases exhibit larger hidden-state magnitudes, and that two contrastive activation directions have higher cosine similarity. Based on these observations, the paper proposes PerFact, which identifies shared attention heads and applies singular value decomposition (SVD) to identify the harmful direction. Experiments on two LLMs show that PerFact achieves the best overall performance across various settings, demonstrating its effectiveness in mitigating PIH.
Critical strengths
The paper addresses an important problem: user-specific context can undermine factual accuracy, an increasingly relevant concern as personalization becomes more prevalent in LLMs. Moreover, the formulation in Equation 7 and the use of SVD to identify harmful activation directions offer a novel methodological perspective. In particular, Equation 7 preserves the interaction between user preferences and answer correctness, providing a meaningful basis for identifying and mitigating personalization-induced hallucinations.
Critical weaknesses
The empirical gains appear marginal. Given the relatively small number of questions in each dataset, the improvements amount to only one additional correct answer on CommonsenseQA and MMLU when using Llama-3.1-8B, raising concerns about the practical significance and robustness of the reported gains.
Moreover, although the authors cite [1], a more comprehensive comparison with this work is needed. Since [1] also investigates personalization-induced hallucination (PIH) and proposes a steering mechanism to mitigate it, the authors should more clearly distinguish PerFact from this prior approach and provide further empirical comparisons to establish its advantages.
[1] When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs, Findings ACL 2026
Questions for authors
-
In the paragraph spanning lines 245–250, could the authors clarify how the probing embeddings are mapped to prediction probabilities? Specifically, is an MLP projection layer trained for this purpose?
-
Could the authors clarify the key differences between the proposed method and [1] and provide empirical comparisons to better establish the advantages of their approach?
-
How is the residual magnitude reported in Figure 4 computed?
-
Are the selected shared heads relevant to each other? I'm curious whether this head configuration changes when you select those heads using only partially sampled questions.
[1] When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs, Findings ACL 2026
Dear Reviewer,
Thank you for your thoughtful review and constructive feedback on our work. We sincerely appreciate the time and care you have devoted to evaluating our paper.
As the rebuttal period closes in approximately four hours, we wanted to follow up on our response. If your schedule permits, we would be grateful to hear whether it addresses your concerns and whether any points would benefit from further clarification. We would be happy to elaborate within the remaining time.
Thank you again for your time and valuable feedback.
Best regards,
The Authors