- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates forgetting mechanisms for the memory of LLM agents. Long-term memory accumulates over the course of a conversation. However, the retrieval of weakly relevant memories can interfere with access to truly useful evidence. Simply deleting memories is problematic, as information that might prove useful in the future could be lost irretrievably.
The paper introduces FOCUS, a reversible forgetting mechanism that partitions memory into an "Active Set" and a "Dormant Set." Each memory entry is assigned an activation value that decays naturally over time. If an entry is retrieved and subsequently utilized, its activation value is reset to its maximum. If it is not used, it is assigned a temporary, progressively diminishing activation value. When an entry’s activation value falls below a specific threshold, it is moved to the Dormant Set. Following each response generation, the LLM evaluates whether the retrieved memories actually contributed to the response. Other components of the framework are based on LightMem.
Experiments conducted on the LoCoMo and Memora long-term conversational memory benchmarks, using the Qwen3-30B and DeepSeek-V4-Flash models, demonstrate that FOCUS outperforms all baselines in terms of accuracy. Ablation studies reveal that removing any component, such as lifecycle management, attribution-based evaluation, reactivation, or specific operational choices, leads to a decline in performance. While the computational cost of maintaining memory is higher than that of LightMem, it remains lower than that of other baselines.
Critical strengths
-
The problem is clearly defined, and the design which decouples the retention of information from its participation in retrieval is sound. FOCUS distinguishes between whether a memory is stored and whether it is directly utilized during retrieval. This approach is both simple and versatile, it can be integrated into other memory systems and is highly practical for long-running, personal-assistant-style agents.
-
The system performs exceptionally well in long-conversation scenarios, supported by robust statistical reporting. Experiments were conducted across two benchmarks with the longest configuration in Memora reaching an average of approximately 31,000 conversational turns and evaluated using two different backbones alongside five representative baselines, ensuring comprehensive coverage. The reported results represent the average of three independent runs, with the appendix providing standard deviations and bootstrap confidence intervals.
-
The engineering design prioritizes efficiency and cost-effectiveness. Group summaries are updated in batches to avoid frequent LLM calls, while the use of two distinct thresholds prevents memories from oscillating back and forth between different sets. These design choices clearly reflect a consideration for real-world deployment scenarios.
Critical weaknesses
-
The claims regarding novelty are overstated. The paper repeatedly asserts that FOCUS is the first long-term memory framework to manage the retrieval space through reversible memory state transitions. However, several cited works already feature similar mechanisms. For instance, MemoryBank employs a forgetting curve with intensity refreshed upon retrieval, while MemoryOS migrates items between hierarchical layers based on their "heat" (popularity/relevance). Furthermore, MemOS includes an archival state within the memory lifecycle from which items can be restored. The construction component draws from LightMem, and the selection of operations follows Mem0. The genuine innovations lie in distinguishing between updates triggered by active usage versus those resulting from simple retrieval, as well as a two-stage retrieval process that utilizes dual thresholds and groups dormant memories into summaries.
-
The primary mechanism proposed in the paper lacks direct empirical validation. There is no direct evidence for the core claim of "reversibility", specifically, no analysis are provided on how many dormant memories are subsequently reactivated, whether these reactivated memories are actually utilized in generating correct responses, nor is there a comparison against a variant where such memories are simply deleted. In the ablation study, the w/o Reactivation condition still retains the retrieval of dormant memories, therefore it does not effectively test the distinction between reversible and irreversible processes. Furthermore, the accuracy of the attribution mechanism which relies entirely on the LLM to determine which memories were utilized remains unverified, even though the entire activation mechanism hinges on these judgments. Additionally, the problem identified in the motivation that weakly relevant memories compete with useful evidence is never directly measured. The paper fails to report metrics such as retrieval precision or recall, leaving it unclear whether FOCUS actually succeeds in mitigating such interference.
If the two points above can be clearly clarified, I might be inclined to accept.
Questions for authors
- Could you provide the results for the hard deletion variant?
- What proportion of dormant memories are reactivated, and of those reactivated, how many are actually put to use?
3 comments
Official Comment by Authors (2/2)
W2. Concerns on empirical validation.
A: Thanks for the suggestion. We complement the existing ablations with direct measurements of reactivation and retrieval, a hard deletion comparison, a human audit of memory use attribution, and an analysis of earlier evidence recovery.
2.1 Reactivation and retrieval. We first measured how often Dormant entries returned to the Active Set and how many returned following actual use. With Qwen as the backbone on Memora Quarterly, we tracked entries marked Dormant at session 500 through session 1500. Of these entries, 36.32% returned to the Active Set at least once. Among the reactivated entries, 47.60% returned following attributed use, corresponding to 17.29% of the original Dormant entries.
To examine whether relevant memories remained retrievable after entering the Dormant Set, we tracked the marked relevant entries during QA evaluation after memory construction. Specifically, we measured source coverage as the percentage whose groups were selected through summary matching, and final recall as the percentage that reached the answering context.
| Benchmark | Eligible questions | Source coverage (%) | Final recall (%) |
|---|---|---|---|
| LoCoMo | 736 | 94.70 | 78.83 |
| Quarterly | 175 | 88.95 | 53.73 |
These results show that FOCUS can recover relevant information through group summaries, even in Quarterly histories spanning nearly 2,000 sessions. However, memory entries differ across methods, so we retain end-to-end Accuracy and FAMA as the primary metrics for system comparison.
2.2 Hard deletion. To test whether retaining forgotten entries benefits later queries, we compared FOCUS with a hard deletion variant that deletes memory entries when their activation meets the forgetting condition, using Qwen on both benchmarks.
| Method | LoCoMo Accuracy | Quarterly FAMA |
|---|---|---|
| Hard deletion | 31.55 | 13.47 |
| FOCUS | 75.97 | 38.50 |
According to the above table, hard deletion exhibits a substantial performance drop relative to FOCUS. This shows that, once deleted, forgotten information is unavailable to later queries, limiting the system’s ability to answer questions that depend on it. We will include these results in the ablation study section of the revised manuscript.
2.3 Use attribution audit. To assess the reliability of memory use attribution, we collected 92 judgments during memory construction, using 12 to establish annotation criteria and the remaining 80 for human auditing. Treating human labels as the reference and “used” as the positive class, precision is computed as TP/(TP+FP) and recall as TP/(TP+FN), where TP denotes correct “used” predictions, FP denotes predictions of “used” for memories labeled unused, and FN denotes predictions of “unused” for memories labeled used.
| Backbone | Precision (%) | Recall (%) |
|---|---|---|
| DeepSeek | 83.6 | 85.9 |
| Qwen | 78.3 | 75.8 |
The human audit indicates that use attribution (with both backbones) provides informative but imperfect feedback. Although memories associated with FN judgments may receive only temporary activation and those associated with FP judgments may enter or remain in the Active Set, subsequent feedback from contextually related interactions can help limit their effects on memory management.
2.4 Recovery of earlier evidence. We further examined whether earlier information remains accessible as subsequent interactions accumulate. With Qwen as the backbone on Memora Quarterly, we marked questions and relevant memory entries whose evidence appeared before session 500, then continued memory construction through session 1500. We report FAMA at three checkpoints and recall of the marked entries at session 1500.
| Method | FAMA: session 500 | FAMA: session 1000 | FAMA: session 1500 | Recall: session 1500 (%) |
|---|---|---|---|---|
| Mem0 | 27.47 | 24.23 | 22.18 | 19.16 |
| LightMem | 22.07 | 16.51 | 15.12 | 18.84 |
| FOCUS | 29.90 | 31.29 | 28.51 | 38.66 |
FOCUS achieves higher FAMA at every checkpoint and higher final recall. Together, these results show that FOCUS preserves access to earlier evidence as history grows, helping sustain answer quality over long conversations.
Q1. Could you provide the results for the hard deletion variant?
A: Thanks. We have explained it in our response to Weakness 2.2.
Q2. What proportion of dormant memories are reactivated, and of those reactivated, how many are actually put to use?
A: Thanks. We have explained it in our response to Weakness 2.1.
Follow-up on Our Rebuttal
Dear Reviewer,
Thank you again for your thoughtful and constructive review. We have provided detailed responses to your comments and updated the manuscript accordingly.
As the rebuttal deadline (November 18, 11:59 PM AoE) is approaching, we would like to kindly follow up and ask whether our responses have addressed your questions and concerns.
We would greatly appreciate any further feedback you may have.
Best regards,
The Authors
Official Comment by Authors (1/2)
We appreciate your careful reading and comments on the contribution and empirical validation of FOCUS. We clarify the novelty claim and present results on reactivation and hard deletion, followed by a human audit of memory use attribution and an analysis of earlier evidence recovery.
W1. Concerns on novelty.
A: Thanks. We agree that the broad “first” claim overstates our contribution to some extent. In this paper, our main contribution is exactly reversible retrieval space management that balances a smaller direct retrieval space with continued access to individual atomic memory entries. Although the proposed FOCUS is somewhat similar to MemoryBank, MemoryOS, and MemOS in memory state management, it has large distinctions from these related methods. Specifically, MemoryBank uses probabilistic forgetting that can delete memories; MemoryOS can evict mid-term segments; MemOS supports archival restoration, but its documented Archived state is excluded from ordinary inference by default and accessed through special retrieval or backtracking. In contrast, FOCUS retains complete ormant entries within the normal retrieval process through group summaries, while allowing them to return to the Active Set. We will narrow the “first” claim and discuss these distinctions in the revised manuscript.
Furthermore, the proposed framework has two main parts: memory extraction and memory management. For memory extraction, we directly adopt LightMem’s buffering and topic segmentation for extraction and draw on Mem0’s explicit operations to incorporate extracted facts into the memory store, without any modifications. In this paper, our novelty actually lies in memory management mechanism, i.e., exploring the lifecycle and retrieval participation of stored entries, by combining activation-driven Active/Dormant transitions, memory use attribution, and retrieval through group summaries. We will revise the manuscript to emphasize our novelty in memory management but not in memory extraction.