- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes as an external memory framework for long-term personalization via LLM agents. maintains an explicit, revisable hypothesis that is grounded to the evidence in the history. It first converts each interaction into compact evidence, uses it to create an evolving hypothesis of the user, revises the hypothesis (updates the current belief state) based on newer information, and finally performs a personalized action after retrieval. does not train a new personalization model from the user sequences; rather, it uses PLLMs to interpret interactions and an explicit memory structure, which helps to store, revise, retrieve, and ignore the beliefs about the user. The central motivation stated is that raw observations of the user's actions do not provide the true underlying interests of the user. Therefore, this interest state remains partially observable by the LLM agent. Results show that completes more tasks successfully that compared baselines.
Critical strengths
1: Explicit decomposition of memory into evidence-gorunded, revisable beliefs for personalized tasks
The main strength I see is the explicit decomposition framed for a personalization process. The separation of raw interaction history from its interpretations: interactions being converted into atomic traceable evidence, and then being organized into dynamic themes. The explicit hypothesis containing proposition, applicability conditions, confidence, supporting and conflicting evidences, is a genuine interesting idea, although not completely new. , already explores the areas of explicit natural-language user preferences, conditional preferences/exceptions, interpretable/editable user model, and concept of belief revision in long-sequence LLM tasks. I still consider this to strength provided that is positioned around them more clearly, since the core formulation of evidence provenance, positive/negative evidence sets, confidence dynamics remain distinct.
2: The structured updating of new evidences
The framework interprets new evidences relative to the existing user hypotheses and performs multiple update actions and so on, which allows the agents to understand and distinguish repetation, obsolete relation, conditional refinement and one-off noise in user interaction history.
Critical weaknesses
1: Problem Framing has scope issue
The problem statement is framed around partially observable user states, non-stationarity of preference evolution, and need for continual revision of user hypotheses. However, in sequential personalization, fundamental idea that user state depends on previous history and the newest interaction, already exists. Recurrent models, transformer-based models such as Mamba4rec, GRU4Rec, LSTUR, NAML, NRMS, already update their internal representation of user as new evidences arrive. More specialized models also explicitly distinguish different temporal components of user interests which are long-term, short-term as well as episodic bursty interests. Hence the novelty seems to be externalization of user state into evidence-grounded hypotheses instead of \textbf{\textit{user understanding should revise the model's belief as new evidences arrive}}.
2: Limited experimental coverage relative to personalization
Vitabench2 is used as evaluation for end-to-end personalized task execution. It evaluates the full downstream action pipeline, and the final success rate establishes the downstream utility of the full memory system, but does not measure the quality of intermediate belief representation itself. However, the question remains to be investigated is
. Another related concern is whether SOTA sequential recommenders perform better than their current performances when explicit evidences are decomposed and then fed to their per-timestep user latent represnetations.
Questions for authors
-
Is it impossible to apply to personalization benchmarks such as MIND news, PENS headline personalization, MovieLens movie prediction, and Amazon product recommendation tasks?
-
Is there a learning of the hypotheses update? Can you provide whether the hypotheses actually provides quality revisions?
-
Can there be standard metric-based evaluations such as NDCG, MRR, HitK?
2 comments
Response: Clarifying Personalization Claims and Evaluation
Thank you for the detailed response.
W1: Problem Framing
I acknowledge the distinction between latent sequential representations and NOBLE's explicit hypotheses with evidence, applicability conditions, confidence, and lifecycle operations. My concern was that updating user states through is already fundamental to sequential personalization. I welcome the authors' commitment to position the contribution specifically around explicit belief revision rather than the broader notion of non-stationary user modeling.
W2: Is Personalization Quality Directly Evaluated?
I recognize that VitaBench evaluates preference-conditioned task success using ground-truth rubrics, not merely successful API execution. The improvements and belief-update ablations are useful evidence of downstream utility. However, they do not directly isolate whether the explicit belief representation produces better personalized selections than existing user-state models.
Consider a hotel-booking example. Historical interactions indicate that a user prefers Marriott hotels, but a recent interaction reveals that proximity to a hospital is currently more important, while budget constraints remain applicable.
Suppose the environment contains:
- Hotel A: Marriott, within budget, far from hospital.
- Hotel B: Non-Marriott, near hospital, within budget.
- Hotel C: Marriott, near hospital, over budget.
Assume the benchmark rubric identifies Hotel B as the valid choice. NOBLE may REFINE the existing preference and help the agent book Hotel B. However, this result does not establish whether a conventional sequential user representation would also select Hotel B.
Suggested Controlled Evaluation
Since VitaBench already provides histories, candidate environments, and preference rubrics, I suggest evaluating candidate-selection correctness independently of agent execution on suitable subtasks.
- Fix identical history , current request , candidate set , metadata, and ground-truth rubric .
- Let a sequential personalization model predict a top-1 candidate ID using its learned user representation.
- Let NOBLE select an ID using its explicit hypotheses and evidence under the same task conditions.
- Deterministically map each predicted ID to its metadata and evaluate both against .
For example, define:
1,&\text{if candidate }i\text{ satisfies }R_t,\\ 0,&\text{otherwise}. \end{cases}$$ Then compare $y_{t,\hat{i}_t}$ across approaches. This requires neither a trained ranking head nor booking-agent execution for the sequential recommender. If complete rankings and relevance labels are available, Hit@K, MRR, or NDCG@K could additionally be computed. I understand that not every VitaBench task reduces to one candidate ID. A controlled subset with independently defined candidate relevance would still be informative. Such an evaluation tests rubric-based personalized selection, not prediction of an independently observed future user choice. ### Correctness of Belief Revision A second unresolved question concerns intermediate hypothesis quality. In the same example, NOBLE might correctly REFINE the preference to: > Generally prefers Marriott, except when current trip-specific constraints take priority. Alternatively, it might incorrectly SUPERSEDE the belief with: > No longer prefers Marriott hotels. Both interpretations might lead to booking Hotel B successfully, yet they imply different user states for subsequent tasks. Therefore, I would appreciate direct evidence concerning: - Correctness and factual support of hypotheses. - Accuracy of applicability conditions. - REFINE versus SUPERSEDE decisions. - Handling temporary exceptions versus lasting changes. - Reliability/calibration of confidence updates. The belief-update ablation establishes the usefulness of revisions for downstream success, but not necessarily the correctness of individual revisions. The authors acknowledge that such direct evaluation has not been conducted. ### Current Assessment I recognize the explicit decomposition and positive VitaBench results as meaningful strengths. The rebuttal substantially clarifies the intended positioning, but the experimental concerns remain: **the proposed belief representation is not directly evaluated for correctness or compared with learned sequential representations on controlled personalized candidate selection**. A focused candidate-level experiment using existing rubrics, together with direct revision-quality analysis, would provide evidence more closely aligned with the claimed mechanism. Broader recommendation or generation benchmarks would further establish transfer, although I recognize their adaptation may exceed the rebuttal scope. I therefore **retain my Weak Reject** at this stage and remain open to reconsideration if additional evidence addresses these concerns.
We thank the reviewer for the constructive comments on NOBLE's positioning and evaluation.
W1
We agree that sequential personalization already models evolving user interests, including long- and short-term components. NOBLE's contribution lies in operationalizing evidence-grounded belief revision as an external memory mechanism for personalized agents. Its update unit is an individual interpretation of the user, with explicit rules for how new evidence reinforces, qualifies, or replaces that interpretation.
A temporary restriction and a lasting preference change can both be recent, yet call for REFINE and SUPERSEDE, respectively. NOBLE distinguishes these updates and links them to evidence. We will center the novelty claim on this revision mechanism and its task-conditioned use, with partial observability and non-stationarity as motivation.
W2
Our experiments evaluate whether the maintained beliefs help an agent perform personalized tasks. The belief-update ablation further isolates the contribution of continual revision while retaining hypothesis construction and the downstream reading components. These results support the practical value of updating user interpretations as evidence accumulates.
Direct evaluation of revision correctness would provide complementary evidence about the intermediate representation. Likewise, feeding decomposed evidence into a sequential recommender is a useful way to examine whether the benefits extend to learned user representations. Our current experiments do not establish gains in next-item prediction or personalized content generation; we will make the scope of the empirical claims explicit.
Question 1
NOBLE can be adapted to these tasks by constructing memory from the available user history and using the retrieved hypotheses and evidence to condition a ranking or generation model. Recommendation requires a candidate-scoring interface, while personalized generation can use the memory as additional context. Item-ID-based histories also require an appropriate representation or metadata interface for evidence extraction. Our current evaluation focuses on personalized agent task execution. We have not evaluated NOBLE on these datasets and consider them valuable extensions for assessing transfer to recommendation and generation.
Question 2
Hypothesis revision occurs at inference time. A pretrained LLM Judge assesses new evidence against existing hypotheses, a Resolver executes the revision, and confidence is updated through a deterministic rule. We do not train a separate update policy on user sequences; the evolving state is the external memory.
The belief-update ablation shows that enabling revision improves downstream task success compared with retaining hypothesis construction without continual updates. It supports the utility of the revisions, while a direct revision-quality assessment would additionally require judgments of factual support, applicability, and whether each change appropriately refines or replaces the previous hypothesis. We have not conducted that direct assessment.
Question 3
We thank the reviewer for this valuable suggestion. Our current memory evaluation focuses on downstream task success and question-answering accuracy, without a dedicated ranking-based evaluation. VitaBench 2.0 likewise does not provide a standardized ranking protocol with candidate sets and relevance annotations for computing NDCG, MRR, or Hit@K. These metrics would be valuable for personalized agents, complementing end-to-end success by assessing how well relevant memories or preferred items are ranked. Establishing the corresponding annotations and evaluation protocol is a useful direction for extending the benchmark.