acceptodds
Under review as a conference paper at ICLR 2027

Infini Memory: Maintainable Topic Documents for Long-Term LLM Agent Memory

Abstract

Large language models (LLMs) enable personal assistant agents that work with users over long-term interactions. These assistants need memory that retains information across conversations and updates it when that information changes. Existing memory systems typically extract and summarize conversations into retrievable memory records. However, this process can fragment related information across records, causing incomplete retrieval and conflicting updates, while repeated summarization progressively discards details. We propose Infini Memory, a long-term memory system inspired by organizing notes around topics while preserving links to their source conversations. It organizes extracted entries into topic documents that serve as persistent units of revision. When new information arrives, the topic-document pipeline revises each selected document using its complete body and the incoming entries. A separate Source-Evidence Layer archives dialogue before extraction and preserves links from entries to their sources as documents evolve. At query time, the system retrieves relevant topic documents and follows source links to recover missing details. Experiments on MemoryAgentBench show that Infini Memory reaches an overall score of 64.7, 19.2 points above the strongest evaluated baseline. On the original LongMemEval-S benchmark, it achieves 96.4% accuracy across six question types.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-term agent memory, Topic documents, Memory consolidation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_U969OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Infini-Memory handles long-term memory for agents in a straightforward manner. It organizes information into topic-based documents, stores recent content in a short-term buffer, and maintains a complete archive of the original conversations. As new information arrives, the system rewrites the relevant documents, dynamically splitting or merging them based on shifts in the subject matter. Each memory segment includes a source tag that links back to the original conversation from which it originated. For retrieval, the paper proposes two approaches, hybrid and agentic, with the "source-aware" method capable of accessing the archived conversations directly when deeper context is required.

Critical strengths
  1. The paper is overall easy to read and understand. Smooth Presentation.
  2. Empirical results are promising. The MemoryAgentBench improvement appears across tasks and reader configurations.
  3. The maintenance ablations show a benefit when answering requires combining updated facts.
Critical weaknesses
  1. There is a lack of a clear correspondence between the 96.4% result on LongMemEval-S and the revision mechanism. While Section 4.2 states that the system extracts entries and repeatedly rewrites the complete topical document, Appendix B.2 notes that the memory used for the "source access" experiments actually consists of clustered raw dialogue turns, not the rewritten text. Furthermore, since the derived documents for positive examples already contain the necessary information, source access primarily serves to improve localization and presentation. Consequently, it remains ambiguous as to which specific memory implementation the 96.4% figure actually corresponds to. The source access experiments merely demonstrate that this retrieval configuration is effective; they fail to establish a definitive link between the memory used for evaluation and the extraction-revision pipeline. Although the ablation study appears to support the claims (showing an increase from 91.8% to 96.4% with the "agentic reader"), it conflates several components, topical structuring, plan-driven updates, and splitting/merging, making it impossible to isolate the specific contribution of full-document rewriting within a fixed topical organization. Given that full-document rewriting is a core contribution, this ambiguity directly affects the interpretation of the headline results.
  2. When the query budget is roughly equivalent, what specific advantages does a maintained memory offer over querying the original source directly? A "source-aware reader" begins by scanning the archives—as shown in Table 6, each query involves reading an average of 47.1 evidence objects, with the answer prompt spanning 15,918 tokens. However, the paper fails to provide the total token count (retrieval plus generation) or an end-to-end cost comparison. The most essential missing control experiment would be to use the same model, the same retrieval budget, and the same context-assembly and answer-prompt structure, but restrict the search solely to the archived conversations. The current "no-maintenance" configuration does not serve as this control, as it still incorporates extraction and "CURRENT" rewriting. While Table 1 notes a shared backbone and ingestion chunk size, this only tells part of the story and does not prove that the computational effort during the query phase is comparable. Since the system is inherently capable of retrieving information by accessing the original archives, a direct "source-only" comparison is necessary to isolate the actual contributions of building and maintaining a memory, specifically regarding its impact on both accuracy and query costs.
Questions for authors
  1. For the original results of LongMemEval-S presented in Tables 2, 3, and 5, which specific memory construction pipeline was used? How does the "clustering of raw dialogue turns" (mentioned in Appendix B.2) relate to the processes described in Section 4.2, namely, the extraction of specific entries and the rewriting of entire topical documents?
  2. Using the same "source-aware reader", but restricting it to query only archived conversations, while keeping the search budget, context window limit, and answer prompt constant, what level of performance can be achieved? Furthermore, how are the denominators and aggregation rules in Table 1 calculated? For instance, in Table 7, DetectiveQA consists of 71 questions, with accuracy as the metric; however, the figures 78.4% and 77.2% do not appear to be simple binary micro-averages based on those 71 questions. Does this represent an average taken across different runs or groups, or does the actual number of evaluated items differ?
Reviewer_QZJ5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Infini Memory, a long-term memory system inspired by organizing notes around topics while preserving links to their source conversations, aiming to address the issues of record fragmentation, incomplete updates, and omitted details that exist in current systems. The core of Infini Memory contains: (1) Topic Documents, which organize extracted information into topic documents, and in the consolidation stage, the complete document body is rewritten together with new entries as persistent revision units; (2) Source-Evidence Layer, which archive original dialogues before extraction, and links to the original sources are retained for each entry, so that the missing context can be restored during retrieval. Infini Memory achieves competitive performance against multiple baselines on the MemoryAgentBench and LongMemEval-S benchmarks.

Critical strengths
  1. This paper proposes a Infini Memory, a long-term memory system inspired by organizing notes around topics while preserving links to their source conversations.
  2. Infini Memory achieves competitive performance against multiple baselines on the MemoryAgentBench and LongMemEval-S benchmarks.
Critical weaknesses
  1. The three core challenges proposed in the paper ( (i) record fragmentation; (ii) incomplete updates; (iii) omitted details ) have been widely discussed in recent Agent Memory research. Additionally, the Source-Evidence Layer in Infini Memory is highly similar to LycheeMemory V2 / PGMem. Finally, the idea of categorization by topic was already mentioned in SeCom and LightMem. The authors need to elaborate on the differences from these works.

  2. Infini Memory continuously rewrites new entries with the "complete document body" through LLM. During this process, does it introduce LLM hallucinations? thereby causing semantic drift and irreversible semantic loss?

  3. Although Infini Memory achieves competitive performance on MemoryAgentBench and LongMemEval-S, it lacks a complete efficiency evaluation, for example (i) lack of latency for retrieval/inference; (ii) lack of Total token cost (memory construction / retrieval / inference), making it difficult to support the feasibility of Infini Memory in real scenarios (e.g., personal assistants). Moreover, in Table 6, the Prompt tokens in the No evidence scenario are 16,683, which is higher than the Prompt tokens (15,918) in the Full setting, presenting a contradiction in the data.

Questions for authors

Please see Weakness

Reviewer_JjfvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Infini Memory, a long-term memory framework for LLM agents that organizes extracted information into topic documents and retains links to original conversations. The method consolidates new entries through topic assignment, full-document revision, and split/merge operations, while a Source-Evidence Layer preserves the original dialogue for retrieval. Experiments on MemoryAgentBench and LongMemEval-S report strong performance, with additional ablations on memory maintenance, source-aware reading, and retrieval strategies.

Critical strengths
  1. The paper is well-organized and easy to follow. It addresses an important problem in long-term agent memory, where related facts may be scattered across records and earlier information may conflict with later updates.
  2. The experimental results are strong across several memory capabilities. Infini Memory-A scores 64.7 on MemoryAgentBench, compared with 45.5 for the strongest evaluated baseline, and the ablations provide useful evidence for the roles of maintenance and source-aware retrieval.
Critical weaknesses
  1. The gains are difficult to separate from the stronger retrieval setting. On LongMemEval-S, source access raises accuracy from 84.0% to 96.4%, while the full reader examines 47.1 archived Evidence objects per question on average in the 50-question efficiency study. The paper lacks a compute-matched baseline that retrieves directly from the original conversations, so the benefit of the proposed memory organization remains unclear.
  2. The maintenance ablation changes several components at once. The “w/o maint.” setting removes topic-document formation, plan-driven updates, Split, and Merge together, making it hard to identify the contribution of full-document revision. The paper also discusses related source-preserving memory systems, such as PGMem and LycheeMemory V2, without including them in the empirical comparison.
Questions for authors
  1. Could the authors compare Infini Memory against a simpler raw-conversation retrieval baseline with the same LLM, archive access, retrieval rounds, and context budget? This would help determine whether the gains come from memory organization or from direct access to source dialogue.
  2. Could the authors provide a more fine-grained ablation separating full-document revision, topic assignment, and Split/Merge? In particular, Infini Memory-A reaches 91.8% without maintenance versus 96.4% with it on LongMemEval-S.
  3. Could the authors report end-to-end query latency, total LLM calls, and token or monetary cost, preferably alongside the evaluated baselines? Table 6 reports 47.1 Evidence reads and a 15,918-token answer prompt on average, but the cost of the complete retrieval process remains unclear.
Reviewer_rmbdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

InfiniMemory maintains topic documents that are revised as new information arrives, alongside an archive of original conversations linked to extracted entries. Splitting and merging adjust document boundaries, while hybrid or agentic readers assemble answer context. The system reports 64.7 on MemoryAgentBench and 96.4% accuracy on LongMemEval-S. Ablations examine maintenance, source access, retrieval, and model choice.

Critical strengths
  • Practical integration of revision and provenance. Maintaining related information together while retaining original dialogue is a useful design for evolving conversational memory.
  • Substantial gains against the evaluated baselines. MemoryAgentBench covers several capabilities, and internal ablations indicate benefits from both maintenance and source access.
Critical weaknesses
  1. The comparisons do not establish superiority over the closest alternatives. The main evaluation includes seven established systems, but omits discussed alternatives addressing thematic consolidation, evolving information, or provenance, including EverMemOS, Hindsight, Zep, PGMem, and LycheeMemory V2. LongMemEval-S primarily compares internal variants. Moreover, equivalent retrieval budgets and end-to-end costs are not established, leaving the contribution of additional query-time processing uncertain.
  2. The central organizational mechanism is insufficiently validated. Topic assignment relies on qualitative LLM instructions without direct evaluation of emerging topics, assignment consistency, or corrections affecting multiple topics. Appendix B.2 also states that the inspected LongMemEval-S store contains clustered raw turns rather than rewritten prose. Together with the absence of a matched archive-only reader, this makes it difficult to isolate the benefit of topic-document revision from improved source access. Fragmentation counts across differently sized record types do not resolve this concern.
Questions for authors
  1. How does the system compare with the closest consolidation and provenance-based alternatives under matched models, retrieval budgets, and total query costs?
  2. What is the performance of an archive-only agentic reader? Which LongMemEval-S experiments use rewritten fact documents versus clustered raw turns?
  3. How are emerging topics and corrections affecting multiple topics handled, and what evidence demonstrates consistent assignment and updating in these cases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.