acceptodds
Under review as a conference paper at ICLR 2027

HorizonBench: Long-Horizon Personalization with Evolving Preferences

Abstract

User preferences change over months of interaction, and a personalized assistant must recognize when a later life event supersedes a preference the user stated earlier. Natural conversations reveal neither the user’s current preferences nor the cause of a change, so this ability cannot be measured on real data. We introduce HorizonBench and its data generator, the first resource that explicitly links a user’s latent preference state to months of conversation and records the life event behind every preference change. The generator produces conversations from sampled state trajectories, tracks when each preference changes separately from when it is expressed, and calibrates how likely a life event is to change preferences, and by how much, against a 13-week longitudinal study with 160 participants. HorizonBench contains 4,245 questions from 360 simulated users, each asking for the response that best fits the user’s current preferences; when a preference has changed, one option reflects its earlier value. All 25 frontier models we evaluate choose this outdated option more often than expected if errors were spread evenly (29.2% of wrong answers against 25%), and the best model reaches 52.8% accuracy. The pattern holds across context lengths and expression styles, including when the preference was expressed shortly before the question, and retrieval and memory methods do not recover the lost accuracy, because the evidence of a change is a life event that never states the new preference. Remembering what a user said and knowing what the user prefers now are therefore separate capabilities, and current models lack the second. HorizonBench supports research on long-context models, memory systems, theory-of-mind reasoning, and user modeling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Horizon Personalization, Preference Evolution, Social Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JD9xOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark for following preferences that change over long conversations. It generates user histories from a structured state representation and uses a human study to set the probability and magnitude of preference changes. Models choose responses that fit the current preference, with outdated preferences included among the wrong answers. The experiments compare models, and retrieval methods.

Critical strengths

The proposed generator makes it possible to change history length and how clearly preferences are expressed while retaining a record of the intended changes. This supports controlled comparisons between different personalization methods.

Critical weaknesses

1. I’m not convinced the main finding is new.

PersonaMem [1] already tests whether models follow current preferences and documents errors where they choose outdated ones. PrefEval [2] and ICF-Bench [3] also test following a newer preference when it conflicts with an earlier one. The paper discusses PersonaMem and PrefEval but does not discuss ICF-Bench. I still don’t see what we learn here beyond the same failure on another dataset. The authors need to explain what their benchmark reveals that these earlier evaluations could not.

2. I don’t see how the human study shows the benchmark is valid.

Whether someone’s preferences change frequently or rarely, the assistant still needs to follow their current preference. The study helps choose how often and how much preferences change in the generated conversations, but it does not establish what someone’s new preference should be after a particular event. The paper acknowledges that these directions of change remain synthetic [4]. I would like evidence that people reading the conversation can actually infer the answer the generator assigns.

3. The model evaluation does not tell us enough about current models.

The evaluation includes GPT-5 and Claude Opus 4.5 but leaves out more recent models [4]. Newer models have shown improvements on long-context tests [5], so their behavior on this benchmark is worth checking. Without that evaluation, I can’t tell whether the paper’s conclusions still hold for current models.

4. The memory evaluation misses important parts of how people actually use assistants.

The paper tests Mem0, but Mem0 is only one way agent systems can manage memory [4]. An agent can update stored notes, revisit earlier conversations, and search again when the first result is insufficient. MemCoE [6] shows benefits from learning how to update memory, while ReFind [7] shows benefits from letting the agent search repeatedly. Claude Code also uses compaction, persistent notes, and retrieval as part of its normal operation [8].

I would like to see whether the findings hold with this kind of memory management before the paper draws broader conclusions about the assistants users interact with.

Reference

[1] Jiang et al. Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale. COLM 2025.

[2] Zhao et al. Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs. ICLR 2025.

[3] Qian et al. Do LLMs Forget What They Should? Evaluating In-Context Forgetting in Large Language Models. ICLR 2026.

[4] Anonymous. HorizonBench: Long-Horizon Personalization with Evolving Preferences. ICLR 2027 submission.

[5] OpenAI. GPT-6 Astra: A New Generation of Intelligence. 2026.

[6] Xu et al. Learning How and What to Memorize: Cognition-Inspired Two-Stage Optimization for Evolving Memory. ACL 2026.

[7] Li et al. When Your Agent Opens the Chat App: Agent-Controlled Search over Raw Chat Logs Rivals Structured Memory. arXiv preprint, 2026.

[8] Anthropic. Effective Context Engineering for AI Agents. 2025.

Questions for authors
  1. What do we learn from this benchmark that we could not learn from earlier preference-update benchmarks?

  2. What does the human calibration add, and can people reading the conversations reliably infer the intended new preferences?

  3. Do these failures still occur with newer models?

  4. Do these findings hold for agents that can update their memory and search the conversation again when needed?

5 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for their review, and for noting that the generator "supports controlled comparisons." We used that control to address each weakness with new experiments on the two newest models, gpt-6-astra (astra), the model the reviewer cites [5], and gemini-4-argon (argon). Brackets are 95% bootstrap intervals.

W1. The finding is new: earlier benchmarks state the change, HorizonBench implies it

"PersonaMem [1] already tests whether models follow current preferences … PrefEval [2] and ICF-Bench [3] also test following a newer preference when it conflicts with an earlier one … I still don't see what we learn here beyond the same failure on another dataset."

The three benchmarks differ from HorizonBench in how the change reaches the model:

BenchmarkHow the change appears
PrefEvalNo change: preferences are static (Table 1; Zhao et al., 2025).
PersonaMemEvents can motivate a change, but the user states the new preference ("I prefer Mediterranean cuisine now"; Jiang et al., 2025).
ICF-BenchAn explicit update reverses an earlier preference.
HorizonBenchA life event implies the change; the user never announces it (§3.2).

What we learn that these benchmarks cannot show (new experiment). On 200 items, we built inputs from each item's own history containing every relevant conversation (median 5.3K tokens), in two versions that differ by one sentence. In the stated version, the user adds: "By the way, after {event}, my preference changed: for {preference}, my {attribute} is now {value}."

  • astra: 49.0% stated vs. 27.0% implied (+22.0 [14.5, 29.5]).
  • argon: 69.0% stated vs. 58.0% implied (+11.0 [3.0, 19.0]).

The stated case is what the three benchmarks measure. The 11–22 point drop when the change is implied is what HorizonBench measures, and it is invisible in the stated setting. Because the earlier value is one of the answer options, choosing it measures this failure directly. The revised Related Work (§6) discusses ICF-Bench, and we thank the reviewer for the pointer.

W2. People infer the designated value on 86.7% of items

"I would like evidence that people reading the conversation can actually infer the answer the generator assigns."

New human study. For 60 items (20 per generator), 3 annotators read the user's earlier value and their turns in the life-event conversation, then chose which of the item's five values describes the user after the event. The majority chose the designated value on 86.7% of items, a single annotator on 85.0% on average, and the majority on 83.3% of the 36 items where the event is the only evidence (Fleiss κ = 0.79; chance 20%). A new option check confirms that each answer option expresses its value: given one option alone, astra and argon name its recorded value for 81.6% / 82.0% of 5,000 options. Together, these show that a reader who infers the change can identify the answer (§4.3, Appendix G).

Human accuracy on full items is not a meaningful reference: each history averages 163K tokens, roughly a 400-page book. PersonaMem, which the reviewer cites, makes the same choice: "we are not asking for human performance on the questions, given the intractability of reading the long contexts" (Jiang et al., 2025).

"The study helps choose how often and how much preferences change … but it does not establish what someone's new preference should be after a particular event."

We agree, and the longitudinal study was not designed to. It calibrates how often and how much preferences change, from 83,024 ratings by 160 people (§3.2, Appendix C). Establishing the new value is the role of the human study above.

AuthorsOpenReview ↗

Rebuttal (2/2)

W3. The newest models still lose accuracy when the change is implied

"The evaluation includes GPT-5 and Claude Opus 4.5 but leaves out more recent models … Newer models have shown improvements on long-context tests [5] … I can't tell whether the paper's conclusions still hold for current models."

We evaluated gpt-6-astra, the model in [5], and gemini-4-argon on the 200 items (% correct):

ModelFull historyChange impliedChange stated
astra [5]28.527.049.0
argon63.558.069.0
  • Both models lose 11–22 points when the change is implied rather than stated, so the conclusion holds for current models.
  • Long-context gains do not address this failure. With only the relevant 5.3K tokens in context, accuracy does not rise (28.5 → 27.0; 63.5 → 58.0). The failure is in inferring the change, which a longer context window does not supply.

W4. A note-taking memory agent does worse, not better

"An agent can update stored notes, revisit earlier conversations, and search again … I would like to see whether the findings hold with this kind of memory management."

  • New experiment: an agent that updates stored notes. On the same 200 items, each model first writes the memory notes it would keep about the user from every relevant conversation, instructed to record any preference that changed and why, then answers from its notes and the current conversation. astra scores 21.5% and argon 42.0%, below reading the same conversations directly (−5.5 [−10.5, −0.5] and −16.0 [−22.5, −9.5]), and their choices of the earlier value rise to 35.0% and 31.5%. Writing the note requires inferring the change, the step models miss.
  • Revisiting and searching again. Any search, single or repeated, aims to assemble the relevant conversations. Given exactly those conversations, models score 27.0% and 58.0%, no higher than with the full history (§5.3, Appendix H).
  • Designing memory agents is out of scope for a benchmark. HorizonBench is released so that designs such as MemCoE and ReFind can be tested on it, and the revised Related Work (§6) discusses both.

Changes in the revised paper: the human study and option check (§4.3, Appendix G); the matched test with the newest models and the note-taking agent (§5.3, §5.4, Appendix H); ICF-Bench, MemCoE and ReFind in Related Work (§6).

We hope this clarifies the contribution. We are happy to answer any further questions and would appreciate it if the reviewer would update their score accordingly.

Dear reviewer JD9x, since the discussion period is coming to an end, we would like to thank you again for your feedback. We believe that we have addressed all your concerns and would love to engage in any further discussion. If there are no further concerns, we would appreciate if you would consider raising your score to reflect the updated understanding of our paper. Thank you so much for your time!

Reviewer_g3TuOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

HorizonBench evaluates personalization under changing user preferences. A synthetic generator maintains a structured user state, constructs event-linked preference transitions, and generates conversations from these states. The benchmark contains 4,245 multiple-choice questions from 360 simulated users. For evolved items, an option represents the user's original preference, allowing the authors to measure selection of an outdated value. The paper reports lower accuracy on evolved preferences and elevated selection of old-value distractors, interpreting these findings as evidence of belief-update failure.

Critical strengths
  1. Long-term personalization requires more than retaining earlier user statements. Explicitly representing preference transitions, expression timestamps, and associated events provides a useful framework for investigating this distinction. Including the original preference among the answer options also enables a more targeted error analysis than aggregate accuracy alone.

  2. The paper evaluates 3 model families and three generators, and includes expression rewrites, a shorter-horizon condition, relaxed filtering, mention-frequency analysis, and retrieval and memory baselines.

Critical weaknesses
  1. The event and preference transition are jointly constructed to satisfy a predetermined change requirement. Section 3.2 and Appendix G.2 indicate that the generator selects preferences to evolve and asks an LLM to produce both a significant event and qualitatively different preference values. The prompt requires every listed preference to have at least one changed attribute, together with an explanation of why the event caused the change.

  2. Ground truth inside the simulator is not necessarily identifiable from the observable history. The generator can know the current state because it assigned that state. A model observing only the conversation may reasonably consider several current preferences possible. The paper acknowledges that histories can be consistent with multiple latent states, but generating a single state does not remove this ambiguity for the observer.

  3. Human validation does not yet resolve the ground-truth concern. Development-round majority accuracy is 63.7% and 56.4%, with moderate agreement. The final annotation pass is more relevant, but Table 12 defines full agreement as three annotators selecting the same answer, rather than explicitly requiring agreement with the designated label. Consequently, unanimous agreement does not by itself establish that the generator's answer is recoverable.

  4. The experiments do not sufficiently isolate belief-update failure from alternative explanations.

Questions for authors
  1. What percentage of final benchmark items receive unanimous human agreement with the designated answer, using only model-visible information? Do the central results hold on this subset?

  2. Can you separate event-only items from items with subsequent expressions of the updated preference, and report human accuracy, model accuracy, and old-value selection for each?

  3. How do models perform when given all relevant evidence, or the current preference explicitly? Can a matched test hold the question and options fixed while changing whether valid update evidence is present?

4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for their precise questions, and for finding that representing transitions, timestamps and events "provides a useful framework" and that the earlier-value option "enables a more targeted error analysis." We ran the studies the reviewer asks for, with the two newest models, gpt-6-astra (astra) and gemini-4-argon (argon), neither of which generated or filtered the benchmark. Brackets are 95% bootstrap intervals.

Summary

  • The ground truth is recoverable. People infer the designated value from the event (86.7% majority, 70.0% unanimous), and each option expresses its value (81.6% / 82.0%).
  • We ran the matched test the reviewer proposed. With the question and options fixed, stating the change adds 22.0 points (astra) and 11.0 (argon).
  • The central result holds where people agree unanimously, for the paper's 25 models and for the new ones.

New studies used below

  • Human study. For 60 items (20 per generator), 3 annotators saw the user's profile, the earlier value of the tested attribute and the user's turns in the life-event conversation, and chose which of the item's five values describes the user after the event. This tests whether the new value follows from the event.
  • Option check. astra and argon saw one option alone, with the attribute and the item's five values but no history, and named its recorded value for 81.6% / 82.0% of 5,000 options (1,000 items; chance 20%). This tests whether each option expresses its value.
  • Matched test. We sampled 200 items (stratified by generator) whose history contains a conversation expressing the earlier value and the life-event conversation. From each item's own history we built inputs that all end with the same test conversation, question and options: change implied (the earlier-value conversations, the life-event conversation and any later ones, in date order), change stated (the same plus one user sentence: "By the way, after {event}, my preference changed: for {preference}, my {attribute} is now {value}"), and the full history.

W1. Readers outside the generator infer the same change

"The event and preference transition are jointly constructed…"

Constructing them jointly is how HorizonBench records the cause of every change, and the link holds for readers outside the generator: given the event, people chose the designated value on 86.7% of items (majority; chance 20%), and on 83.3% of the 36 items where the event is the only evidence.

W2 and Q1. The designated value is identifiable, and the results hold where people agree unanimously

"What percentage of final benchmark items receive unanimous human agreement with the designated answer…? Do the central results hold on this subset?"

  • Human accuracy on full items is not the right test. Each history averages 163K tokens, roughly a 400-page book per question, which no annotator can read and keep track of. PersonaMem states this directly: "we are not asking for human performance on the questions, given the intractability of reading the long contexts" (Jiang et al., 2025). PrefEval and LongMemEval likewise report no human accuracy on their full histories. What must be established is that each item's answer is correct, and it is when the new value follows from the event (human study) and the options express their values (option check).
  • The item asks for the best of five options, not the exact latent state. The four alternatives are chosen to contrast maximally with the designated value (§4.1). Several states may fit a history; the item requires only that the designated value fit better than the alternatives.
  • People identify it. All three annotators chose the designated value on 70.0% of items and the majority on 86.7% (Fleiss κ = 0.79).
  • The central result holds on those 42 unanimous items. The paper's 25 models answer 29.9% correctly, and 29.8% of their wrong answers choose the earlier value, above 25% for 22 of 25 models. This matches the full benchmark (29.2%).

The new models on the same 42 items (each cell: % correct / % of all answers choosing the earlier value):

ModelFull historyChange stated
astra33.3 / 38.157.1 / 21.4
argon76.2 / 19.083.3 / 9.5
AuthorsOpenReview ↗

Rebuttal (2/2)

W3. Agreement with the designated label is what the new study measures

"Table 12 defines full agreement as three annotators selecting the same answer… unanimous agreement does not by itself establish that the generator's answer is recoverable."

We agree that agreement among annotators alone does not establish recoverability. Table 12's agreement is among annotators, and the revised caption states this definition clearly. Human accuracy against the designated label is reported by the new annotation study: all three annotators chose it on 70.0% of items and the majority on 86.7%. The development rounds (63.7%, 56.4%) annotated earlier generator versions, whose problems, such as options that paraphrased each other, informed changes to the generator before release (App. F). The revised §4.3 rests on the new study.

W4. The submission already tests the main alternatives, and we add new controls

"The experiments do not sufficiently isolate belief-update failure from alternative explanations."

The reviewer's own summary lists these tests among the paper's strengths: "expression rewrites, a shorter-horizon condition, relaxed filtering, mention-frequency analysis, and retrieval and memory baselines." Each targets an alternative; the matched test adds direct controls:

AlternativeTestResult
Long contextShort-horizon condition (§5.3, Table 3)Shorter histories widen the gap to −5.7, negative for 25/25 models.
Failed retrievalRetrieval baselines (§5.4, Table 4); new matched test (§5.3, App. H)Retrieval lowers accuracy for every model. With only the relevant 5.3K tokens, accuracy does not rise (28.5 → 27.0; 63.5 → 58.0).
Mention frequencyMention counts (§5.3, Table 9)The new value is mentioned more in two-thirds of histories; models still choose the earlier one.
Reliance on explicit statementsExplicit vs. implicit rewrites, same 981 items (§5.3, Table 3)Implicit expression widens the gap to −3.6.
Filter artifactRelaxed filter (§5.3, Table 3); new: dropping the 4 filter modelsDistractor rate above 25% for 25/25 and 21/21 models.
Generator familyNo own-family advantage (§4.3); new: matched test on Gemini-3-flash items (App. H)Stating the change adds +24.0 (astra) and +16.0 (argon) on Gemini-3-flash items.
Missing or unclear evidenceNew: implied vs. stated, one sentence apart (§5.3, App. H)Stating the change adds +22.0 [14.5, 29.5] and +11.0 [3.0, 19.0].
The stated sentence helps only because it repeats the option's value labelNew: the same sentence paraphrased, never naming the value (App. H)The gain remains (astra +16.9 [9.7, 24.6]); the paraphrase is within 3.6 points of the stated sentence for both models.
Any added sentence at that point helpsNew: an irrelevant sentence in the same place (App. H)astra −0.5 [−4.1, 2.6]; argon +4.6 [0.0, 9.2].
Indistinct optionsNew option check (§4.3, App. G)Models name each option's recorded value for 81.6% / 82.0% of 5,000 options.

Q2. On event-only items, people still infer the value, and models lose the most

"Can you separate event-only items from items with subsequent expressions of the updated preference, and report human accuracy, model accuracy, and old-value selection for each?"

People (majority): 83.3% on event-only items (36), 91.7% with a later expression (24). Models on the matched test (each cell: % correct / % of all answers choosing the earlier value):

Itemsastra impliedastra statedargon impliedargon stated
Event only (109)20.2 / 37.646.8 / 22.951.4 / 23.963.3 / 19.3
Later expression (91)35.2 / 38.551.6 / 26.465.9 / 23.175.8 / 14.3

Q3. Yes: with the question and options fixed, stating the change adds 11–22 points

"How do models perform when given all relevant evidence, or the current preference explicitly? Can a matched test hold the question and options fixed while changing whether valid update evidence is present?"

We ran the test the reviewer proposes. Each cell: % correct / % of all answers choosing the earlier value.

InputMedian tokensastraargon
Full history117K28.5 / 29.563.5 / 19.5
Change implied (all relevant evidence)5.3K27.0 / 38.058.0 / 23.5
Change stated (current preference explicit)5.3K49.0 / 24.569.0 / 17.0

Implied → stated: +22.0 [14.5, 29.5] and +11.0 [3.0, 19.0]. This difference is the cost of inferring the change.

Changes in the revised paper: the human study, option check and unanimous-item analysis (§4.3, Appendix G); the matched test and its controls (§5.3, Appendix H); Table 12's agreement defined in its caption.

We hope this clarifies the contribution. We are happy to answer any further questions and would appreciate it if the reviewer would update their score accordingly.

Dear reviewer g3Tu, since the discussion period is coming to an end, we would like to thank you again for your feedback. We believe that we have addressed all your concerns and would love to engage in any further discussion. If there are no further concerns, we would appreciate if you would consider raising your score to reflect the updated understanding of our paper. Thank you so much for your time!

Thanks. The response addressed many of my concerns. However, considering the overall quality of the paper, I have decided to maintain my original score.

Reviewer_Je1vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that users’ preferences change over months of interaction, often because of life events that never state the new preference, and that an assistant must update its belief about the user and not only retrieve earlier statements. Because the true preference state cannot be observed in natural conversation, the authors build a generator that samples a sequence of user preference states from a structured graph. It produces months of conversation from those states and records the life event behind every change, separately from when the preference was expressed. They calibrate the probability and size of event-linked change against a longitudinal survey of participants.

From this generator they build HorizonBench, a multiple-choice benchmark. Each question asks which assistant response best fits the user’s current preferences, and when a preference has changed, one option reflects the earlier value (the pre-evolution distractor). Items are filtered to those that several LLMs answer wrongly without the history. The authors evaluate 25 models with the full history, varying context horizon, expression explicitness and filter strictness, and also test retrieval and memory methods. They report that models pick the outdated option more than a uniform-error reference predicts and do worse on changed than unchanged preferences. They also report that retrieval and memory methods do not help. They conclude that remembering what a user said and knowing what the user prefers now are separate capabilities, and that current models lack the second.

Critical strengths

The generator is a useful design that natural data cannot provide. Because it separates when a preference changes from when it is expressed, and records the event behind each change, it supports controlled variants (expression explicitness, time since expression, filter threshold) and allows retrieval to be scored against the recorded event conversation, as in the BM25 analysis in Section 5.4. The release of the generator, the mental state graphs and the human longitudinal dataset (83,024 ratings) is a substantial resource for work on evolving user state.

The authors test several alternative explanations instead of reporting a single headline number. They check context length with the short-horizon variant, mention frequency (Table 9), expression explicitness and filter strictness (Table 3). They also check for position bias and for a generator-family advantage. They state the limits of the filter openly: it shares four models with the evaluated panel, and it makes accuracy fall below chance for ten models. Evaluating 25 models plus five open-weight models, with bootstrap intervals, makes the observed pattern hard to attribute to one model family.

Critical weaknesses

It is unclear that the items are reliably answerable from the history, which undercuts the conclusion that models lack the ability. In the development rounds, human majority-vote accuracy was 56–64% (Fleiss κ 0.37–0.51), and all three annotators missed 19% of Round 2 items. Appendix F calls this “partial recoverability together with substantial ambiguity”, while Section 4.3 says items are recoverable. In the benchmark-wide pass, the three annotators gave the same answer on 673 of 4,235 items and three different answers on 1,305 (Table 12). The paper reports no human accuracy against the designated answer on the released benchmark. Model accuracy is about 24% at every agreement level, which the paper reads as item clarity not mattering, but it is also consistent with model answers not depending on answerability. The Section 2 assumption that options differ only in the value of dimension k is not checked directly, since Table 13 rates only situational fit, flow realism and persona consistency.

The headline effect is small and not isolated as a belief-update failure. The mean distractor rate is 29.2% against a 25% reference, so about 70% of wrong answers go to other options. The reference assumes uniform errors, although the earlier value is the only option guaranteed to appear in the history (Section 3.4) and the other three are LLM-proposed (Section 4.1). The average evolved–static gap is −2.7 points. By my calculation, with 2,484 evolved and 1,761 static items, one model’s gap has a standard error of about 1.4 points. Static accuracy is itself low (13.3% for gpt-5-mini, 15.4% for gpt-5). The sign and Wilcoxon tests are over the 25 per-model values, although all models answer the same items.

Several numbers do not seem to reconcile:

  1. Section 5.2 gives a mean distractor rate of 29.2% and a maximum of 37.8%, while the same paragraph reports 37.0% and 39.9% for OpenAI family means.
  2. Table 3 gives 38.2% for the neutral condition, and Table 12 gives 34.8–42.9% by agreement level.
  3. Table 12’s accuracies (about 24%) are about two points below Table 7’s averages (26.3%) for nearly the same items. In Table 3 the distractor rate is 38.2% (neutral), 49.7% (explicit) and 58.8% (implicit), but the text describes only the explicit-to-implicit rise.

The claim that remembering and knowing are separate capabilities is not tested. No condition places the recorded event conversation and the earlier preference expression in context, so retrieval failure and inference failure are not separated. The retrieval baselines return ten turns. BM25 retrieves the event conversation less often than a random draw (18.5% vs 22.1%). Mem0 is tested with one reader whose full-history accuracy (19.3%) is near the 20% chance level. The human calibration sets two parameters from participants’ ratings of their own everyday preferences, while the benchmark’s preferences concern assistant response style. The event rate ranges from 0.07 to 0.70 across thresholds (Table 6). The “first resource” claim depends on the feature columns chosen in Table 1, and there is no empirical comparison with concurrent work such as PERMA, Memora and STALE.

Questions for authors

What is human accuracy against the designated answer on the released benchmark, overall and by agreement level? Is the “agreement” in Table 12 agreement on that answer? Given that annotators gave the same answer on 673 of 4,235 items and three different answers on 1,305, how should model accuracy be read relative to humans, and why are the all-different items kept? Section 3.2 says life events leave preferences unchanged 85% of the time. How can a model tell, from the history, an event that changed the tested preference from one that did not? Does the history contain such unchanged-preference events for the tested preference? No condition guarantees that the evidence of a change is in context, so retrieval failure and inference failure are not separated. The memory comparison also uses one reader whose full-history accuracy is 19.3% (chance is 20%), and the retrieval baselines return ten turns. How do you support the claim that remembering what a user said and knowing the user’s current preference are separate capabilities, and that retrieval and memory methods cannot help? The distractor rates do not reconcile. Section 5.2 gives a mean of 29.2% and a maximum of 37.8%, while the same paragraph gives 37.0% and 39.9% for OpenAI family means (the OpenAI bars in Figure 3 appear to be roughly 30%). Table 3 gives 38.2% for the neutral condition, and Table 12 gives 34.8–42.9% by agreement level. Table 12’s accuracies (about 24%) are also about two points below Table 7’s averages (26.3%) for nearly the same items. What are the correct values, and what are the distractor rate and the evolved–static gap per generator? The neutral, explicit and implicit conditions in Table 3 use only the o3-generated subset. The evolved and static items are not described as matched pairs, and the sign and Wilcoxon tests are computed over the 25 per-model values although all models answer the same items. The binomial tests do not mention clustering by user. In Table 3 the distractor rate is 38.2% (neutral), 49.7% (explicit) and 58.8% (implicit), but the text describes only the explicit-to-implicit rise, and the conditions use different item sets (for example 233 evolved items in the short-horizon condition). Which differences can be attributed to the controlled factor, and how reliable are the per-model and family-level conclusions? Four of the five filter models are among the evaluated models, and ten models score below the 20% chance level. How should absolute accuracies and rankings be interpreted, particularly for the models that also served as filters? The calibration uses participants’ ratings of their own everyday preferences, while benchmark preferences concern assistant response style. How does the calibration carry over, how sensitive are results to the threshold (Table 6) and to p_evo, and what role does it play given the enriched evolved share?

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for their careful reading, for checking our numbers, and for calling the generator "a useful design that natural data cannot provide." We answer each concern with new experiments on the two newest models, gpt-6-astra (astra) and gemini-4-argon (argon), neither of which generated or filtered the benchmark; brackets are 95% bootstrap intervals.

Summary

  • Items are answerable. People infer the designated value from the life event (86.7%), and each option expresses its value (81.6% / 82.0%).
  • The failure is a failure to update. With items, options and evidence fixed, stating the change adds 22.0 points (astra) and 11.0 (argon).
  • Remembering is not knowing. Exactly the relevant conversations do not raise accuracy; a note-taking agent does worse.
  • The numbers reconcile once matched to their item sets.

W1. Items are answerable: the change follows from the event, and options express their values

"What is human accuracy against the designated answer on the released benchmark? … how should model accuracy be read relative to humans?"

Human accuracy on full items is not a meaningful reference for this benchmark. Each history averages 163K tokens, roughly a 400-page book per question. PersonaMem makes the same choice: "we are not asking for human performance on the questions, given the intractability of reading the long contexts" (Jiang et al., 2025).

An answer is correct when the new value follows from the life event and the correct option expresses it. We tested both.

  1. The change follows from the event (new human study). For 60 items (20 per generator), 3 annotators saw the user's profile, the earlier value of the tested attribute and the user's turns in the life-event conversation, and chose which of the item's five values describes the user after the event. The majority chose the designated value on 86.7% of items, a single annotator on 85.0% on average, and all three on 70.0% (Fleiss κ = 0.79; chance 20%). On the 36 items where the event is the only evidence of the change, the majority chose it on 83.3%.
  2. Each option expresses its value (new option check). astra and argon saw one option alone, with the attribute and the item's five values but no history, and named its recorded value for 81.6% / 82.0% of 5,000 options (1,000 items; chance 20%). This checks the §2 assumption directly.

Models fall short of people. Given the same evidence on the 60 items (the earlier-value and life-event conversations), argon chooses the designated response on 65.0%, gemini-3.1-pro on 38.3% and astra on 36.7%, against 86.7% for people. On the 42 items all three annotators answered correctly, the paper's 25 models give the earlier value on 29.8% of their wrong answers (22 of 25 above 25%), as on the full benchmark (29.2%).

The development rounds (56–64%, App. F) annotated earlier generator versions; their findings informed changes before release, and the revised §4.3 rests on the new study.

W2. The effect is reliable, and it locates failure to update

"The headline effect is small and not isolated as a belief-update failure… The reference assumes uniform errors."

Reliable. With the reviewer's SE of 1.4 per model, the −2.7 gap is about 2 SE for one model, and it is negative for all 25 models. Resampling the 346 users across all models, which accounts for shared items: gap [−4.2, −1.1], distractor rate 29.2% [26.9, 31.4].

The earlier value belongs in the history. The earlier value is the only wrong option guaranteed to appear in the history by design: it is what the user said before the change. A model tracking the user should treat it as superseded; choosing it is the failure being measured.

New matched test: the earlier value is in context either way, and models still fail to update. For 200 items (stratified by generator), we built inputs from each item's own history that all end with the same test conversation, question and options:

  • Change implied: the conversations expressing the earlier value, the life-event conversation and any later conversations expressing the new value, in date order.
  • Change stated: the same, plus one user sentence at the end of the life-event conversation: "By the way, after {event}, my preference changed: for {preference}, my {attribute} is now {value}."
  • Full history: the benchmark input.

Cells: % correct / % of all answers choosing the earlier value.

InputMedian tokensastraargon
Full history117K28.5 / 29.563.5 / 19.5
Change implied5.3K27.0 / 38.058.0 / 23.5
Change stated5.3K49.0 / 24.569.0 / 17.0

The implied and stated inputs differ by one sentence; both contain the earlier value and the life event. Stating the change adds +22.0 [14.5, 29.5] (astra) and +11.0 [3.0, 19.0] (argon). This difference is the belief-update failure.

AuthorsOpenReview ↗

Rebuttal (2/2)

W3. Remembering and knowing are separate capabilities

"How do you support the claim that remembering … and knowing … are separate capabilities, and that retrieval and memory methods cannot help?"

  • In the paper. The explicit and implicit rewrites of the same 981 items keep the items, events and conversations fixed and change only how preferences are expressed. The gap widens from −1.8 (explicit) to −2.9 (neutral) to −3.6 (implicit) as the change must be inferred from subtler cues (Table 3).
  • Matched test. The change-implied input holds every relevant conversation and nothing else. It scores no higher than the full history (27.0 vs. 28.5; 58.0 vs. 63.5), while stating the change adds 11–22 points. With the evidence in hand, models fail to infer the change.
  • Memory. We thank the reviewer for noting that BM25 and Mem0 miss the evidence. A note-taking agent does worse (new): when astra and argon first write notes about the user from every relevant conversation, told to record any preference that changed, and then answer from those notes, they score 21.5% and 42.0%, below the 27.0% and 58.0% they reach reading the conversations. Designing memory systems is out of scope for a benchmark; HorizonBench is released so that such designs can be tested on it.

The numbers reconcile once matched to their item sets

Reviewer's comparisonExplanation
OpenAI 37.0 / 39.9% vs. max 37.8% (§5.2)Those means came from the o3-generated items. Full benchmark: 28.7 / 29.9%, as in Fig. 3, now corrected in §5.2.
Table 3 (38.2%) vs. Table 12 (34.8–42.9%)Table 3 uses the o3 items; Table 12 came from an earlier evaluation pass. Recomputed on the final results: accuracy 25.8 / 26.6 / 25.9% and distractor rate 30.0 / 30.4 / 26.8% (full / majority / none), in line with Table 7. Model accuracy is similar at every level.
Is Table 12's agreement with the designated answer?No, it is among annotators; the revised caption states this. Human accuracy against the designated answer comes from the new study in W1.
Neutral → explicit → implicit (Table 3)Same 981 items. Explicit wording states the earlier value plainly too, so it becomes easier to find (38.2 → 49.7%); implicit wording makes the change harder to infer (58.8%).

Per generator (o3 / Claude-sonnet-4.5 / Gemini-3-flash): distractor rate 38.2 / 25.7 / 24.6%, gap −2.9 / −2.8 / −2.5.

Other questions

  • "Does the history contain such unchanged-preference events for the tested preference?" Yes. Most life events in a history leave the tested preference unchanged, so the model must infer which preferences each event affected.
  • "How should absolute accuracies and rankings be interpreted, particularly for the models that also served as filters?" The filter keeps items that five models answer wrongly without the history, so a model's history-free guess points to a wrong option by selection. Scores below 20%, including Mem0's reader at 19.3%, follow from this (§5.2), and 20% is not the chance level for these models. We therefore read the within-model gap and distractor rate. Without the 4 filter models, the gap is negative and the distractor rate above 25% for 21/21 models.
  • "How does the calibration carry over…?" The calibration makes the benchmark's rates realistic: how often and how much preferences change after a life event, measured from people's own ratings, which the generator applies to response-style preferences. The threshold in Table 6 changes the estimated rate of change, not how an item is scored: the gap and distractor rate are computed on each changed item.
  • PERMA, Memora, STALE. All three are cited and compared in Table 1, §6 and Appendix A. HorizonBench is the only one that offers the exact earlier value as an answer and records the life event behind each change. Their scores and the distractor rate measure different quantities and cannot be put on one scale.

Changes in the revised paper: the human study, option check and per-generator statistics (§4.3, Appendix G); the matched test and note-taking agent (§5.3, §5.4, Appendix H); user-clustered intervals and corrected OpenAI means (§5.2); Table 12 recomputed, with its agreement defined in the caption.

We hope this clarifies the contribution. We are happy to answer any further questions and would appreciate it if the reviewer would update their score accordingly.

Dear reviewer Je1v, since the discussion period is coming to an end, we would like to thank you again for your feedback. We believe that we have addressed all your concerns and would love to engage in any further discussion. If there are no further concerns, we would appreciate if you would consider raising your score to reflect the updated understanding of our paper. Thank you so much for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.