- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HABITUSMEM to address the problem of long-term user memory in personalized agents. The core argument is that, rather than treating memory as a compression of historical conversations, it should be viewed as a belief about the user’s latent state, with the system deciding what information to retain, update, or retract based on the current request. The authors further introduce structured fields such as claim, evidence, scope, and status, and train a memory writer through supervised fine-tuning. The method is evaluated on PersonaMem-v2 and VitaBench 2.0.
Critical strengths
- The paper does not simply treat personalized memory as a compression of historical information. Instead, it emphasizes that memory should serve the current user request and attempts to distinguish among claim, evidence, scope, and status. This structured representation is intuitively useful for handling user preferences, changing user states, and potentially conflicting information, and it aligns well with the practical requirements of long-term memory agents.
- The paper analyzes the effects of static writer, current request conditioning, and different training tasks, and separately investigates forgetting/retraction scenarios. These experiments provide some discussion of the method’s applicability and failure modes.
Critical weaknesses
- There is a clear gap between the Bayesian formulation and the actual algorithm. The paper proposes a Bayesian belief and a risk-based objective, but the actual implementation does not estimate the posterior or explicitly optimize this objective. Instead, it trains a teacher-supervised memory writer. Therefore, it is currently unclear whether the Bayesian formulation itself contributes to the performance gains.
- The contribution of teacher-generated supervision is difficult to disentangle. The training targets are primarily generated by a teacher, and their factual correctness is not systematically verified. Therefore, the observed performance gains may primarily come from teacher distillation, request conditioning, or the structured schema, rather than from the proposed Bayesian principle itself.
- The forgetting/retraction capability remains relatively weak. On forgetting queries, HABITUSMEM significantly underperforms full context and some baselines, and it performs particularly poorly on active near-duplicate contradictions. This suggests that the proposed status/retraction representation does not yet adequately resolve the problems of stale memories and contradictions.
Questions for authors
- If the Bayesian formulation were completely removed while keeping the same backbone, training data, teacher targets, request conditioning, and memory schema, how would the performance of HABITUSMEM change?
- How is the quality of the teacher-generated targets verified? The training targets in the paper are primarily produced through teacher generation rather than being subjected to human or automated factual-consistency verification. Has the authors’ evaluation checked for hallucinations, incorrect preferences, incorrect scopes, or incorrect retractions in the teacher-generated memories? If the teacher targets contain systematic errors, would the student model simply inherit those errors?
8 comments
Response to Reviewer uD1c (2/2)
Weakness 3: Forgetting and retraction
Our claim is narrower than complete forgetting: an explicit status makes a retraction representable, auditable, and repairable, but it is not by itself sufficient for a frozen agent to obey it. The evidence shows that the status representation works and that the remaining errors arise where the agent consumes it.
The writer records the retraction.
| Check | Result |
|---|---|
PersonaMem-v2 ask_to_forget queries whose memory contains a status: retracted record | 99.3% of 1,048 |
| Controlled probes whose memory keeps the retraction | 10/10 |
| Controlled retraction probes answered correctly, Qwen | 1/10 |
| Controlled retraction probes answered correctly, DeepSeek | 2/10 |
The retraction is present, yet the agent often does not act on it. This is a memory–agent interface problem, not evidence that the status is absent or meaningless.
The remaining errors concentrate in two interface patterns:
| Pattern | Frequency | Accuracy with the pattern | Accuracy without it |
|---|---|---|---|
| The retraction names only an abstract topic, not the revoked item | 85.7% of retraction records | 0.453 | 0.740 |
| An active near-duplicate of the revoked preference remains | 12.4% of memories | 0.262 | 0.526 |
The frozen agent is sensitive to the precision of the record: it often fails to apply an abstract retraction, and a stale active duplicate can override the retracted one.
The interface is repairable without retraining the agent. We edited the stored memories and re-answered the same queries with the same agents:
| Memory | All forgetting queries, Qwen | All forgetting queries, DeepSeek | Active near-duplicate, Qwen | Active near-duplicate, DeepSeek |
|---|---|---|---|---|
| Stored memory | 0.466 | 0.287 | 0.122 | 0.122 |
| + revoked item named | 0.735 | 0.499 | 0.537 | 0.195 |
| + conflicting active record removed | 0.741 | 0.520 | 0.878 | 0.561 |
The same agents honor a precise retraction, and the status and scope fields make the repair a local edit to the record.
Question 2: Quality of teacher-generated targets
Three properties of the pipeline limit how much the result depends on teacher errors.
- The decisive record comes from annotations. In every direct-writing target, the record for the preference that a request depends on is taken from the benchmark's annotation, including whether it is active, superseded, or retracted. The remaining records of the target come from the cold-start writer, which was trained on teacher memories, and the auxiliary targets come from the parent writer.
- The evaluation is end to end on held-out personas. Errors inherited from the teacher would lower accuracy on the 5,000 test queries. Training on teacher-written memories alone (Trained, static) gives 0.406 and 0.335, close to the untrained writer, so the gains do not come from reproducing teacher memories (Weakness 2 above).
- The writer's outputs are checked on the fields the reviewer lists. In the controlled probes (App. D.1), manual inspection confirms that all ten gift memories attribute the item to the gift recipient rather than to the user, and the full writer records the conditional scope and the retraction in all ten memories and the supersession in nine or ten.
The probes and the end-to-end results indicate that teacher errors do not dominate the writer's behavior on these benchmarks.
We hope that our responses have addressed your concerns, and we would be grateful if you would consider raising your score.
Followup on Rebuttal
Dear Reviewer;
Dear Reviewer; Thank you again for taking the time to review our responses. As the rebuttal period will be ending soon, we would greatly appreciate it if you could kindly revisit our responses and let us know whether any concerns or questions remain unaddressed.
We would be more than happy to provide further clarification or additional explanations if anything is still unclear. If our responses and the additional experiments have adequately addressed your concerns, we would also sincerely appreciate your reconsidering the overall score in light of the new evidence. Thank you again for your time and consideration.
Thanks
Response to Reviewer uD1c (1/2)
Weakness 1 and Question 1: Contribution of the Bayesian formulation
We agree that the implementation does not estimate the posterior or optimize the risk directly. The revision removes "Bayesian" from the title and presents the formulation as the design rationale (General Response, Part 2).
Question 1. The formulation is not a module of the trained system. If it were removed while the backbone, training data, teacher targets, request conditioning, and memory schema all stayed the same, the trained writer and its performance would be unchanged. The formulation contributes by motivating two of the components the question holds fixed, request conditioning and the schema, and Tables G1 and G2 of the General Response ablate each separately. We agree with the reviewer that the gains come from these components rather than from Bayesian inference, and the revision states this.
Weakness 2: Teacher distillation, request conditioning, and schema
The revision separates these three factors (Tables 4 and 5).
Request conditioning and schema. Removing the request from the trained writer at inference lowers accuracy by 13.7 and 14.5 points (Table G1). Removing the record fields and retraining end to end lowers it by 1.3 and 1.9 points (Table G2).
Teacher distillation. The training lineage separates training on teacher-written memories from learning to use the request (PersonaMem-v2, Table 5):
Training on teacher-written memories without the request, or adding the request without training, each raises accuracy by at most 2.3 points. The large gain appears only when training teaches the writer to use the request. Distillation from the teacher alone therefore does not explain the result.