- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a benchmark for following preferences that change over long conversations. It generates user histories from a structured state representation and uses a human study to set the probability and magnitude of preference changes. Models choose responses that fit the current preference, with outdated preferences included among the wrong answers. The experiments compare models, and retrieval methods.
Critical strengths
The proposed generator makes it possible to change history length and how clearly preferences are expressed while retaining a record of the intended changes. This supports controlled comparisons between different personalization methods.
Critical weaknesses
1. I’m not convinced the main finding is new.
PersonaMem [1] already tests whether models follow current preferences and documents errors where they choose outdated ones. PrefEval [2] and ICF-Bench [3] also test following a newer preference when it conflicts with an earlier one. The paper discusses PersonaMem and PrefEval but does not discuss ICF-Bench. I still don’t see what we learn here beyond the same failure on another dataset. The authors need to explain what their benchmark reveals that these earlier evaluations could not.
2. I don’t see how the human study shows the benchmark is valid.
Whether someone’s preferences change frequently or rarely, the assistant still needs to follow their current preference. The study helps choose how often and how much preferences change in the generated conversations, but it does not establish what someone’s new preference should be after a particular event. The paper acknowledges that these directions of change remain synthetic [4]. I would like evidence that people reading the conversation can actually infer the answer the generator assigns.
3. The model evaluation does not tell us enough about current models.
The evaluation includes GPT-5 and Claude Opus 4.5 but leaves out more recent models [4]. Newer models have shown improvements on long-context tests [5], so their behavior on this benchmark is worth checking. Without that evaluation, I can’t tell whether the paper’s conclusions still hold for current models.
4. The memory evaluation misses important parts of how people actually use assistants.
The paper tests Mem0, but Mem0 is only one way agent systems can manage memory [4]. An agent can update stored notes, revisit earlier conversations, and search again when the first result is insufficient. MemCoE [6] shows benefits from learning how to update memory, while ReFind [7] shows benefits from letting the agent search repeatedly. Claude Code also uses compaction, persistent notes, and retrieval as part of its normal operation [8].
I would like to see whether the findings hold with this kind of memory management before the paper draws broader conclusions about the assistants users interact with.
Reference
[1] Jiang et al. Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale. COLM 2025.
[2] Zhao et al. Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs. ICLR 2025.
[3] Qian et al. Do LLMs Forget What They Should? Evaluating In-Context Forgetting in Large Language Models. ICLR 2026.
[4] Anonymous. HorizonBench: Long-Horizon Personalization with Evolving Preferences. ICLR 2027 submission.
[5] OpenAI. GPT-6 Astra: A New Generation of Intelligence. 2026.
[6] Xu et al. Learning How and What to Memorize: Cognition-Inspired Two-Stage Optimization for Evolving Memory. ACL 2026.
[7] Li et al. When Your Agent Opens the Chat App: Agent-Controlled Search over Raw Chat Logs Rivals Structured Memory. arXiv preprint, 2026.
[8] Anthropic. Effective Context Engineering for AI Agents. 2025.
Questions for authors
-
What do we learn from this benchmark that we could not learn from earlier preference-update benchmarks?
-
What does the human calibration add, and can people reading the conversations reliably infer the intended new preferences?
-
Do these failures still occur with newer models?
-
Do these findings hold for agents that can update their memory and search the conversation again when needed?
5 comments
Rebuttal (2/2)
W3. The newest models still lose accuracy when the change is implied
"The evaluation includes GPT-5 and Claude Opus 4.5 but leaves out more recent models … Newer models have shown improvements on long-context tests [5] … I can't tell whether the paper's conclusions still hold for current models."
We evaluated gpt-6-astra, the model in [5], and gemini-4-argon on the 200 items (% correct):
| Model | Full history | Change implied | Change stated |
|---|---|---|---|
| astra [5] | 28.5 | 27.0 | 49.0 |
| argon | 63.5 | 58.0 | 69.0 |
- Both models lose 11–22 points when the change is implied rather than stated, so the conclusion holds for current models.
- Long-context gains do not address this failure. With only the relevant 5.3K tokens in context, accuracy does not rise (28.5 → 27.0; 63.5 → 58.0). The failure is in inferring the change, which a longer context window does not supply.
W4. A note-taking memory agent does worse, not better
"An agent can update stored notes, revisit earlier conversations, and search again … I would like to see whether the findings hold with this kind of memory management."
- New experiment: an agent that updates stored notes. On the same 200 items, each model first writes the memory notes it would keep about the user from every relevant conversation, instructed to record any preference that changed and why, then answers from its notes and the current conversation. astra scores 21.5% and argon 42.0%, below reading the same conversations directly (−5.5 [−10.5, −0.5] and −16.0 [−22.5, −9.5]), and their choices of the earlier value rise to 35.0% and 31.5%. Writing the note requires inferring the change, the step models miss.
- Revisiting and searching again. Any search, single or repeated, aims to assemble the relevant conversations. Given exactly those conversations, models score 27.0% and 58.0%, no higher than with the full history (§5.3, Appendix H).
- Designing memory agents is out of scope for a benchmark. HorizonBench is released so that designs such as MemCoE and ReFind can be tested on it, and the revised Related Work (§6) discusses both.
Changes in the revised paper: the human study and option check (§4.3, Appendix G); the matched test with the newest models and the note-taking agent (§5.3, §5.4, Appendix H); ICF-Bench, MemCoE and ReFind in Related Work (§6).
We hope this clarifies the contribution. We are happy to answer any further questions and would appreciate it if the reviewer would update their score accordingly.
Dear reviewer JD9x, since the discussion period is coming to an end, we would like to thank you again for your feedback. We believe that we have addressed all your concerns and would love to engage in any further discussion. If there are no further concerns, we would appreciate if you would consider raising your score to reflect the updated understanding of our paper. Thank you so much for your time!
Rebuttal (1/2)
We thank the reviewer for their review, and for noting that the generator "supports controlled comparisons." We used that control to address each weakness with new experiments on the two newest models, gpt-6-astra (astra), the model the reviewer cites [5], and gemini-4-argon (argon). Brackets are 95% bootstrap intervals.
W1. The finding is new: earlier benchmarks state the change, HorizonBench implies it
The three benchmarks differ from HorizonBench in how the change reaches the model:
What we learn that these benchmarks cannot show (new experiment). On 200 items, we built inputs from each item's own history containing every relevant conversation (median 5.3K tokens), in two versions that differ by one sentence. In the stated version, the user adds: "By the way, after {event}, my preference changed: for {preference}, my {attribute} is now {value}."
The stated case is what the three benchmarks measure. The 11–22 point drop when the change is implied is what HorizonBench measures, and it is invisible in the stated setting. Because the earlier value is one of the answer options, choosing it measures this failure directly. The revised Related Work (§6) discusses ICF-Bench, and we thank the reviewer for the pointer.
W2. People infer the designated value on 86.7% of items
New human study. For 60 items (20 per generator), 3 annotators read the user's earlier value and their turns in the life-event conversation, then chose which of the item's five values describes the user after the event. The majority chose the designated value on 86.7% of items, a single annotator on 85.0% on average, and the majority on 83.3% of the 36 items where the event is the only evidence (Fleiss κ = 0.79; chance 20%). A new option check confirms that each answer option expresses its value: given one option alone, astra and argon name its recorded value for 81.6% / 82.0% of 5,000 options. Together, these show that a reader who infers the change can identify the answer (§4.3, Appendix G).
Human accuracy on full items is not a meaningful reference: each history averages 163K tokens, roughly a 400-page book. PersonaMem, which the reviewer cites, makes the same choice: "we are not asking for human performance on the questions, given the intractability of reading the long contexts" (Jiang et al., 2025).
We agree, and the longitudinal study was not designed to. It calibrates how often and how much preferences change, from 83,024 ratings by 160 people (§3.2, Appendix C). Establishing the new value is the role of the human study above.