- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper addresses over personalization problem of LLMs, and gives various analysis to conclude that warmth is the key to the over personalization problem. Then it identifies dpo stage and reward model as the main stage of over personalization, and propose a method to mitigate it with sft/dpo on the curated data.
Critical strengths
- The paper addresses and gives detailed analysis on the over-personalization problem, which is an important but unrecognized problem of the field.
- The analysis of the dpo stage contributing to most of the over personalization problem (presented with CIR scores) is the first to give a well defined analysis on per training step, to my knowledge.
Critical weaknesses
-
The overall presentation of the paper needs improvement, especially the coherence of its analysis. Sections 4–6 discuss various aspects of the over personalization problem and conclude warmth as the key element. This is more of a behavioral observation than a fundamental analysis (as also stated in Section 6), which weakens the significance of the warmth sensitivity results in Section 7. Also, the warmth problem of Sections 4–6 does not connect to Sections 8–9, where the reward-model probing and the proposed fix do not target warmth, making the transition unnatural. Additionally, the analyses in Sections 4–5 are mostly expected behaviors (zero injection w/o memory, more injection on open-ended tasks, compliance with explicit instructions). These make it difficult to identify the core contribution of this paper.
-
The novelty of the proposed method is limited. Training with DPO on curated preference data is a widely used pipeline, and the data curation procedure here does not introduce a novel component either. Especially, the core idea of training a model to disregard irrelevant context has already been studied in [1], where the main difference lies only in the domain of the prepended content (user persona / documents). Also, as shown in Table 7, the supervised warm-up accounts for most of the CIR reduction, which suggests that the SFT proposed in [1] is enough to mitigate this problem. While this is understandable since the paper weighs more on its analysis, as stated above, the flow from the analysis to the method is unnatural, requiring a more careful logical formulation for an ICLR paper.
[1] Making Retrieval-Augmented Language Models Robust to Irrelevant Context, Yoran et al., ICLR 2024, arXiv:2310.01558.
Questions for authors
See weaknesses.
5 comments
Response to Reviewer 3c32 (Part 2 of 2)
Training contribution and comparison with prior work
The novelty of the proposed method is limited. Training with DPO on curated preference data is a widely used pipeline, and the data curation procedure here does not introduce a novel component either. Especially, the core idea of training a model to disregard irrelevant context has already been studied in [1], where the main difference lies only in the domain of the prepended content (user persona / documents). Also, as shown in Table 7, the supervised warm-up accounts for most of the CIR reduction, which suggests that the SFT proposed in [1] is enough to mitigate this problem. While this is understandable since the paper weighs more on its analysis, as stated above, the flow from the analysis to the method is unnatural, requiring a more careful logical formulation for an ICLR paper.
Response:
We thank the reviewer for highlighting this relevant work. We agree that training models on mixed relevant/irrelevant contexts is not new, and our mitigation shares this principle with Yoran et al. (2024). However, our fundamental distinction is not between retrieved documents and user memories, but between two different failures of context utilization: being misled by irrelevant evidence versus being incentivized to use information that should remain unused. The former primarily concerns factual robustness; the latter concerns how alignment shapes what models choose to include in their responses.
Specifically, the differences are threefold:
(1) Factual contamination vs. discretionary personalization. Yoran et al. study how irrelevant evidence degrades QA correctness. In contrast, our failure can occur even when the answer remains factually correct: personal information unnecessarily influences examples, framing, or content selection. A model can therefore be robust under conventional QA accuracy while still exhibiting substantial personalization overreach. This explains why our phenomenon concentrates on open-ended tasks and requires a separate metric (CIR).
(2) Evidential relevance vs. conversational appropriateness. In RAG, relevance concerns whether retrieved evidence supports an answer. In our setting, memories are accurate and may even be topically relevant, yet their inclusion can still be unwarranted. Indeed, relevance filtering only reduces Gemini's CIR from 29.0% to 25.6%. Knowing that information is relevant does not establish that it should be expressed. This distinction makes selective personalization a different problem from evidence filtering.
(3) Insufficient robustness vs. preference-shaped utilization. Yoran et al. primarily address models' susceptibility to irrelevant retrieval. We investigate an additional mechanism: alignment may actively reinforce unnecessary context use. CIR increases across all five SFT–DPO comparisons, while controlled DPO experiments reproduce this direction. More importantly, warmth increases memory injection even under explicit relevance constraints, although warmth is logically unrelated to whether memory is needed. Nine reward models further exhibit memory-conditioned preferences for persona-using responses. Together, these results suggest that personalization can become a rewarded response behavior, rather than merely an accidental consequence of irrelevant context exposure.
Regarding mitigation, we agree that the general training recipe overlaps with Yoran et al., and we do not claim algorithmic novelty for SFT/DPO. We also acknowledge that Table 7 shows most CIR reduction comes from supervised warm-up, with DPO providing a smaller additional improvement. However, Section 9 serves a different scientific purpose: testing whether this preference-shaped behavior can be reversed without eliminating appropriate memory use. Our training explicitly distinguishes neutral tasks, where memory should remain unused, from memory-required tasks, where it should be applied correctly. The resulting reductions in CIR, alongside largely preserved memory-required accuracy and general capabilities, demonstrate that restraint and useful personalization can be jointly learned. Importantly, we do not infer the origin of overreach from mitigation success alone; evidence for preference-related amplification comes primarily from Sections 6–8 and Appendix E.
In summary, Yoran et al. establish how models can become robust to irrelevant evidence; our work investigates why aligned models may favor unnecessary personalization even without factual errors. Our novelty lies in identifying this accuracy-invisible failure and tracing its relationship to preference alignment, not in the fine-tuning algorithm.
Revision. We cite and discuss Yoran et al. and related studies in Related Work and clarify Section 9's role as a reversibility test.
[1] Yoran, Wolfson, Ram, and Berant. Making Retrieval-Augmented Language Models Robust to Irrelevant Context. ICLR, 2024.
Conclusion
Thank you for raising these concerns about the presentation and logical flow; we agree that they are well founded. In the updated PDF, we have clarified how the analyses lead to the training intervention and how these parts jointly support our core contribution. We have also provided a more careful comparison with Yoran et al. and related work to clarify the connections and distinctions. We hope these revisions make the reasoning and contribution clearer and meet the standard of rigor you rightly expect of an ICLR paper.
Thank you again for recognizing the value of studying personalization overreach. As personal agents become increasingly common, we hope our work will encourage the community to join us in giving this problem greater attention.
Response to Reviewer 3c32 (Part 1 of 2)
W1: Coherence of the analysis and the core contribution
Response:
Thank you for your constructive feedback and for recognizing the importance of personalization overreach and our analysis across training stages. We appreciate your suggestions on improving the coherence of the paper and clarifying how the different analyses contribute to the central argument. We agree that these connections could have been presented more clearly. Following your suggestions, we have revised the Introduction, reorganized the behavioral analyses, improved the transitions between sections, and updated Figure 2 to illustrate the overall structure of the paper.
These revisions aim to establish a clearer progression through three connected questions:
(1) What drives unnecessary memory use?
We have reorganized the original Sections 4–5 into a unified analysis of the memory-utilization gap. The behavioral experiments now follow a connected line of reasoning. We first establish the prevalence and task dependence of personalization overreach, then examine how models respond to memory relevance, forget instructions, and changes in available personal attributes. Together, these observations suggest that models can recognize distinctions among memory attributes and respond to relevance constraints, yet still incorporate personal information unnecessarily. We have clarified how these behavioral findings motivate our central question: how models decide whether to use available personal information in their responses.
(2) How might alignment reinforce this behavior?
We have clarified the role of warmth in the analysis and its connection to the subsequent alignment experiments. Warmth provides a behavioral probe for examining memory utilization: encouraging warmer responses also increases unnecessary memory use, revealing a coupling between conversational warmth and personalization. This observation motivates our investigation of how alignment training may reinforce this behavior. We then examine alignment checkpoints to identify how overreach and its warmth dependence change across training stages, followed by reward-model probing to investigate whether preference signals favor responses that incorporate available personal information. These analyses address different aspects of the same question: how preference optimization may strengthen the tendency to use memory unnecessarily. We have made this connection explicit in the revised section transitions.
(3) Can this behavior be changed while preserving warmth and useful personalization?
We have also clarified how the preceding analyses motivate our mitigation approach. The findings suggest that unnecessary personalization is associated with a learned tendency to incorporate available memory into responses. This motivates selective-memory training, which teaches models when personal information should be used and when it should remain unused. Our training distinguishes neutral tasks, where memory should not influence the answer, from memory-required tasks, where memory should be applied correctly. We then evaluate whether unnecessary personalization decreases while warmth and useful memory utilization are retained. This intervention allows us to examine whether the identified behavior can be changed without broadly suppressing personalization.
Together, these revisions establish a clearer progression: characterizing unnecessary memory use, identifying warmth as a behavioral probe, examining how preference optimization reinforces this tendency, and testing whether selective-memory training can mitigate it. We have made this progression explicit in the revised Introduction, Figure 2, and the connections between sections.
Our core contribution is studying and characterizing personalization overreach across models, examining how warmth and preference optimization contribute to it, and exploring how to reduce unnecessary memory use while retaining useful personalization.