acceptodds
Under review as a conference paper at ICLR 2027

EnSIMem: Entity-Structured Indexing for Long-Term Agent Memory

Abstract

An agent that interacts with users over long periods must recall facts, preferences, events, and changes from a continuously growing interaction history. Existing memory systems often compress interactions into generic summaries or retrieve anonymous text chunks, making it difficult for an agent to identify the correct entity, property, and supporting evidence. We present EnSIMem, an entity-structured long-term memory architecture for an agent. During offline construction, the system organizes interactions into theme-coherent episodes and builds dialogue-grounded index entries of the form . Each entry preserves its source turns, temporal information, and available multimodal fields. During online interaction, the agent's request is decomposed into evidence requirements whose properties are aligned with the memory index. Entity-property lookup and adaptive retrieval then collect the evidence needed for point, temporal, compositional, and aggregation reasoning. The agent generates its response from the preserved source evidence rather than from lossy memory summaries. On long-term agent-memory benchmarks, EnSIMem achieves high answer accuracy while maintaining compact, evidence-focused contexts. These results show that entity-structured indexing and episode-level provenance provide a reliable foundation for long-term memory in agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-term Agent Memory, Entity-structured Indexing, Episodic Memory, Evidence-grounded Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_E7MVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EnSIMem is a training-free memory pipeline for long-term conversational agents. Offline, an LLM splits each session into contiguous, theme-coherent episodes and extracts [entity][type][property:value] records, each linked to its source turns. Online, an LLM planner turns the query into evidence requirements and assigns a reasoning type. Retrieval combines structured entity–property matching with dense fallback and stops once the requirements are covered. GPT-4.1-mini then answers from the expanded original episodes, so the records serve as addresses rather than as evidence. The paper reports 90.6% on LoCoMo (MEMORA(P): 86.3%) and 92.8% on LongMemEval (87.4%). It adds subset ablations on episode granularity, property granularity, structured vs. dense retrieval, and adaptive vs. fixed top-k. Online latency is 14.4s, against 5.70s for MEMORA.

Critical strengths

Strong, protocol-checked headline results. On LoCoMo the method gains +4.3pp overall over MEMORA(P), including +11.4pp on open-domain and +10.0pp on multi-hop. It also beats Full Context (82.5%) while using fewer context tokens (7,470). The authors re-ran MEMORA(P) locally and obtained 86.23% against the published 86.3% (Sec. 4.1), which supports comparability with that baseline. The two judges agree closely (App. D, r > 0.93). With 1,540 questions, the main gap is well above sampling noise.

Inspectable, training-free design. Separating localization (records) from interpretation (source episodes) produces traceable answers (App. B). The design is a reasonable principle for practical memory systems.

Critical weaknesses

The central mechanisms are not causally isolated. All ablations use an undisclosed "fixed LoCoMo subset." The reported accuracies (96.59 / 94.32 / 90.91 / 86.36%) correspond to 85 / 83 / 80 / 76 out of 88 questions. The granularity effects therefore rest on 2–5 questions, and the adaptive budget beats fixed top-8 by one question (Fig. 3). The subset is also clearly easier than the full set (96.6% vs. 90.6%). More importantly, the two claims the paper rests on are never ablated: (a) answering from expanded episodes versus from records alone (Sec. 3.1, "address, not substitute"), and (b) the requirement-aware planner, which accounts for 10.3s of the 14.4s latency. It is also unclear whether the dense-only variant in Table 4 keeps the planner and the adaptive budget. Finally, the extraction prompt (App. E.1) hard-codes the predicates "research, attend, read, paint, support," which match the conv-26 case studies, and there is no development/test split on a 10-conversation benchmark.

The headline comparisons are not controlled, and Table 2 is inconsistent. All baseline rows are transcribed from Memora, and the paper does not state that they share the GPT-4.1-mini answer model or the same question set. On LongMemEval, every transcribed row reconciles exactly with the LongMemEval-S category sizes (30/56/133/133/78/70) and their weighted average. EnSIMem's rows do not: 91.7% (SS-assistant), 89.3% / 89.1% (temporal), 93.0% / 91.4% (multi-session), and 96.1% (knowledge update) are unattainable fractions at those sizes. The category-weighted average is 92.6%, not the reported 92.8%. App. D's max |Δ| = 3.37pp on SS-preference is also not a multiple of 1/30. This points to a different question set or a different aggregation, so the +5.4pp claim cannot be verified as a like-for-like comparison.

Questions for authors

Which LongMemEval variant did you use, and how many questions per category? How is "Average" computed? Please reconcile Table 2 with the standard category sizes, including how abstention items are handled.

How large is the ablation subset, how was it selected, and was it used during prompt development? Can you rerun the ablations on full LoCoMo and add two variants: (a) answering from records only, and (b) no planner, retrieving with the raw query at a matched budget?

Were the transcribed baselines run with GPT-4.1-mini as the answer model? Can you re-run at least Full Context and one or two memory baselines inside your own pipeline?

What is the offline construction cost (LLM calls, tokens, wall-clock time), especially for LongMemEval's per-question haystacks? How is the index updated when new sessions arrive?

10 comments
AuthorsOpenReview ↗

Response to Reviewer E7MV: Previous Ablation Design and Experimental Coverage

Thank you for your careful review.

Our responses are structured into five sections:

  1. Previous Ablation Design and Experimental Coverage
  2. New Ablations and Prompt Development
  3. LongMemEval-S denominators and aggregation
  4. Controlled Baseline Comparisons
  5. Offline Construction and Incremental Updates

This is Section One: Previous Ablation Design and Experimental Coverage

Thanks for your detailed questions/suggestions on performance study.

W1 + Q2. General comments on previous ablation design and experimental coverage.

We expanded the property-granularity and retrieval-budget experiments from 88 questions to all 347 Category 1–4 questions from conv-48 and conv-49. We now explicitly report the subset’s composition, selection criterion, and prompt-development history, together with exact correct counts for each condition. These experiments retain the same LLMs, embedding configuration, judge, and execution environment as the main evaluation. Below, we explain the subset selection and present the results for property specificity and retrieval budget.

W1.1. “All ablations use an undisclosed 'fixed LoCoMo subset.'” (Q2.1. “How large is the ablation subset, how was it selected, and was it used during prompt development?”)

The original subset contained 88 questions from conv-26. This time we expanded the test size to all 347 questions from conv-48 and conv-49. We now explicitly show the conversations, question counts, selection criterion, and development history.

EvaluationWhere subset fromNo. of QuestionsPurpose
Original ablationsconv-26 subset88Property granularity, fixed retrieval budgets, and dense candidate retrieval
New repeated ablationsconv-48 and conv-49347Property granularity, fixed retrieval budgets, and dense candidate retrieval
New core ablationsAll ten conversations1540Records-only answering and no planner retrieval

We selected conv-48 and conv-49 of LoCoMo because their pooled full-EnSIMem accuracy was closest to the full-benchmark accuracy (Please find the table below for specific numbers). We included every Category 1–4 questions in both conversations, without filtering individual questions by correctness. These 347 queries were not used during prompt development.

Evaluation SetCorrect / TotalFull-EnSIMem Accuracy
conv-48172/19190.05%
conv-49135/15686.54%
conv-48 and conv-49 combined307/34788.47%
All ten conversations1361/154088.38%

W1.2. “The reported accuracies (96.59 / 94.32 / 90.91 / 86.36%) correspond to 85 / 83 / 80 / 76 out of 88 questions.” “The subset is also clearly easier than the full set (96.6% vs. 90.6%).”

We agree that the original subset was not big. Therefore, we use a larger subset containing 347 questions from conv-48 and conv-49 for the repeated ablations, and for the two new core variants we used all 1540 questions in testing. This time the subset is not easier, the full EnSIMem score on the full benchmark is very similar to the score on this subset (88.38% vs 88.47%).

W1.3. “The granularity effects therefore rest on 2–5 questions, and the adaptive budget beats fixed top-8 by one question (Fig. 3).”

We now report the new scores for the new testing on 347 questions from conv-48 and conv-49. Property granularity refers to the specificity of the property labels in the EnSI records and corresponding query-plan fields. The source episodes, record identities, entities, values, and supporting evidence are retained. This experiment does not change episode boundaries.

Property-Label GranularityCorrect / TotalAccuracy
EnSIMem (Broad)307/34788.47%
Fine288/34783.00%
Broad+303/34787.32%
Very broad295/34785.01%

The difference between the default EnSIMem setting (Broad) and Broad+ is four questions out of 347. The gap increases to 12 for Very broad and 19 for Fine. Note that Broad+ means the granularity is slightly broader than the default setting.

For the retrieval budget, we evaluated six fixed numbers of episodes. Each condition uses a prefix of the same ranked candidate list.

top-kCorrect / TotalAccuracy
top-1210/34760.52%
top-3276/34779.54%
top-5286/34782.42%
top-8290/34783.57%
top-15297/34785.59%
top-20305/34787.90%

Within this sweep, increasing the cap from top-1 to top-20 yields 95 additional correct answers. The gap between top-8 and our full EnSIMem is 17 questions.

AuthorsOpenReview ↗

Response to Reviewer E7MV: New Ablations and Prompt Development

Thanks for your detailed review.

Our responses are structured into five sections:

  1. Previous Ablation Design and Experimental Coverage
  2. New Ablations and Prompt Development
  3. LongMemEval-S denominators and aggregation
  4. Controlled Baseline Comparisons
  5. Offline Construction and Incremental Updates

This is Section Two: New Ablations and Prompt Development.

Thank you for your comments about doing more ablations.

W1 + Q2. General comments on new ablations and prompt development

We added records-only answering and single-key retrieval without multi-hop planning on all 1,540 Category 1–4 questions across the 10 LoCoMo conversations. These experiments examine the roles of source-episode evidence and planner-guided retrieval. We also evaluated dense-only candidate retrieval on the 347 questions from conv-48 and conv-49, retaining all the other components to examine the contribution of our EnSI index. Shared components use the same experimental settings as the main evaluation. Finally, we removed the specified prompt examples, rebuilt the index, and reran the full LoCoMo evaluation. Below, we specify each intervention and report its results.

W1.4. “The central mechanisms are not causally isolated.” “More importantly, the two claims the paper rests on are never ablated: (a) answering from expanded episodes versus from records alone (Sec. 3.1, "address, not substitute"), and (b) the requirement-aware planner, which accounts for 10.3s of the 14.4s latency.” (Q2.2. “Can you rerun the ablations on full LoCoMo and add two variants: (a) answering from records only, and (b) no planner, retrieving with the raw query at a matched budget?”)

We completed the following experiments on all 1,540 Category 1–4 questions, and we report their scores now.

ConditionCorrect / TotalAccuracy
Full EnSIMem1361/154088.38%
Records-only answering1284/154083.38%
No planner retrieval191/154012.40%

For records-only answering, we reused the source run’s selected episode identities and their order. We supplied the generation with the extracted records associated with those episodes, excluding original episode text. We retained all the other steps. This directly tests the information available to the generation when source episodes are replaced by their extracted records.

For No planner retrieval, we replaced multi-hop planning with one LLM-extracted structured key, retained all the other steps. This variant directly tests the importance of our planner.

The low score of the variant without planner indicates the importance of planner-guided evidence retrieval. EnSIMem’s planner decomposes a question into explicit evidence requirements and coordinates retrieval across intermediate entities and multiple sources. Removing this process substantially weakens the system’s ability to assemble the evidence needed for answering. These results support the value of requirement-aware planning and its associated retrieval mechanisms.

W1.5. “It is also unclear whether the dense-only variant in Table 4 keeps the planner and the adaptive budget.”

The experiment is specifically a dense-only candidate-retrieval ablation. We did not use the EnSI index in our retrieval. This examines the benefit of our EnSI index. We calculate the embedding of the whole episode and match it with the embedding of the decomposed sub-questions from the planner.

It achieved 276/347 correct, or 79.54%, compared with the full EnSIMem reference of 307/347, or 88.47%.

This experiment retains the planner and all other components and features. It shows the importance of using the EnSI index.

W1.6. “Finally, the extraction prompt (App. E.1) hard-codes the predicates "research, attend, read, paint, support," which match the conv-26 case studies, and there is no development/test split on a 10-conversation benchmark.”

We removed the benchmark-matching examples from the extraction prompt, rebuilt the memory index, and reran EnSIMem across all ten conversations. All the experiments are using the same setting, the same LLM backbone (GPT-4.1-mini), and on the same 1540 questions from LoCoMo. We also rerun Memora (S) and Memora (P) for reference. We now report the scores.

MethodCorrect / TotalAccuracy
EnSIMem1361/154088.38%
Memora (S), local rerun1320/154085.71%
Memora (P), local rerun1321/154085.78%

The revised-prompt result remains above both Memora reruns. This supports the performance of the revised pipeline without relying on the criticized examples.

Prompt revision and development/test separation are distinct issues. We confirm that the 347-query ablation subset was not used for prompt development.

AuthorsOpenReview ↗

Response to Reviewer E7MV: LongMemEval-S denominators and aggregation

Thank you for your careful and detailed review.

Our responses are structured into five sections:

  1. Previous Ablation Design and Experimental Coverage
  2. New Ablations and Prompt Development
  3. LongMemEval-S denominators and aggregation
  4. Controlled Baseline Comparisons
  5. Offline Construction and Incremental Updates

This is Section Three: LongMemEval-S denominators and aggregation.

Thank you for checking the reported numbers carefully.

W2 + Q1. General comments on LongMemEval-S denominators and aggregation.

We now report the reran evaluation on all 500 LongMemEval-S questions and report exact numerators and denominators.

W2.2. “On LongMemEval, every transcribed row reconciles exactly with the LongMemEval-S category sizes (30/56/133/133/78/70) and their weighted average. EnSIMem's rows do not: 91.7% (SS-assistant), 89.3% / 89.1% (temporal), 93.0% / 91.4% (multi-session), and 96.1% (knowledge update) are unattainable fractions at those sizes.” (Q1.1. “Which LongMemEval variant did you use, and how many questions per category?”)

The evaluation uses LongMemEval-S, with all 500 questions, including abstention items. GPT-4.1-mini is the backbone LLM. The category denominators match those identified in the review. We now report the scores.

CategoryCorrectTotalAccuracy
Single-Session Preference253083.33%
Single-Session Assistant525692.86%
Temporal Reasoning11713387.97%
Multi-Session11513386.47%
Knowledge Update727892.31%
Single-Session User7070100.00%
Overall45150090.20%

Each percentage is computed from the corresponding integer counts. These are new, reconciled results.

W2.3. “The category-weighted average is 92.6%, not the reported 92.8%.” (Q1.2. “How is "Average" computed?” Q1.3. “Please reconcile Table 2 with the standard category sizes, including how abstention items are handled.”)

“Average” is micro-averaged accuracy. We sum the correct answers across categories and divide by the total number of evaluated questions.

For example, Accuracy = (25 + 52 + 117 + 115 + 72 + 70) / (30 + 56 + 133 + 133 + 78 + 70) = 451/500 = 90.20%.

Abstention questions remain in their original categories and in the 500-question denominator. They receive credit when the model correctly abstains. They are not excluded from the overall result.

Reviewer_aFEvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes EnSIMem, an entity-structured long-term memory framework for LLM agents. It organizes conversations into theme-coherent episodes, constructs entity-property indices linked to original dialogue evidence, and employs requirement-aware adaptive retrieval. Experiments on LoCoMo and LongMemEval demonstrate strong performance over existing memory systems.

Critical strengths
  • The separation of structured indexing and original evidence preservation enables precise retrieval without losing contextual details.
  • EnSIMem achieves 90.6% on LoCoMo and 92.8% on LongMemEval, outperforming competitive baselines.
  • Clear presentation. The paper is well organized and makes it easy for the reader to quickly grasp the main idea and method.
Critical weaknesses
  • The core ideas of episodic memory organization, structured indexing, and evidence-aware retrieval have been extensively explored in recent works, such as RippleMem [1], E-mem [2], SimpleMem [3], and, in the related procedural-memory setting, HiSkill [4]. The proposed framework appears largely incremental, and its unique technical contributions are not sufficiently distinguished from existing approaches.

  • Despite substantial overlap with recent memory architectures, the paper does not compare against RippleMem, E-mem, or SimpleMem. These omissions weaken the empirical evidence for the claimed superiority of EnSIMem, especially given that several of these methods are evaluated on similar benchmarks.

  • Although structured retrieval improves memory access, it remains fundamentally retrieval-based and may struggle with complex aggregation, logical constraints, and dynamic state management. Alternative paradigms, such as code-based executable memory [5], could potentially provide stronger reasoning capabilities.

  • The experiments are restricted to conversational memory benchmarks. It remains unclear whether the proposed memory mechanism improves actual agent performance in complex, long-horizon tasks. Evaluations in more realistic interactive environments, such as Terminal-Bench, would be more meaningful.

  • EnSIMem requires 14.4 seconds per query compared to MEMORA's 5.7 seconds, limiting its practicality for interactive applications.

[1] Ji, Jingbo, et al. "RippleMem: From Isolated Retrieval to Associative Recollection for Long-Term Agent Memory." arXiv preprint arXiv:2608.13334 (2026).

[2] Wang, Kaixiang, et al. "E-mem: Multi-agent based episodic context reconstruction for llm agent memory." ICML 2026.

[3] Liu, Jiaqi, et al. "Simplemem: Efficient lifelong memory for llm agents." ICML 2026.

[4] Hao, Yu, et al. "Hiskill: Empowering llm agents with hierarchical skill graphs." arXiv preprint arXiv:2607.25853 (2026).

[5] Li, Bojie. "User as code: Executable memory for personalized agents." arXiv preprint arXiv:2606.16707 (2026).

Questions for authors

See weaknesses above

4 comments
AuthorsOpenReview ↗

Additional Experimental Results on Performance Comparison with New Baseline Methods

W2. Missing comparisons with RippleMem, E-mem, and SimpleMem. Thank you for recommending these additional baselines. We have now completed local evaluations of SimpleMem and E-mem using their official implementations, together with the baselines listed below. For RippleMem, we could not obtain an official public implementation and therefore could not include it in the local reruns. All methods were evaluated in our environment on the same 1,540 Category 1–4 questions across all ten LoCoMo conversations, using GPT-4.1-mini for answering and GPT-4o-mini for judging. We report exact correct counts alongside accuracy.

MethodPublished Accuracy from MemoraRerun AccuracyCorrect / Total
Full Context82.50%83.25%1282/1540
RAG63.30%64.03%986/1540
HippoRAG47.10%50.39%776/1540
Mem065.30%57.60%887/1540
LangMem73.40%52.66%811/1540
SimpleMem—85.26%1313/1540
E-mem—81.17%1250/1540
Memora (S)84.90%85.71%1320/1540
Memora (P)86.30%85.78%1321/1540
EnSIMem—88.38%1361/1540

The published column provides reference values. Our conclusions below use the local rerun results under the shared evaluation protocol.

Memora (P) remains the strongest baseline among the methods evaluated locally, achieving 85.78%, compared with 85.26% for SimpleMem and 81.17% for E-mem. This provides direct experimental support for our choice of Memora as a competitive baseline.

EnSIMem also outperforms both newly added methods directly. It answers 48 more questions correctly than SimpleMem, improving accuracy by 3.12 percentage points, and 111 more than E-mem, improving accuracy by 7.21 percentage points. Against Memora (P), EnSIMem answers 40 additional questions correctly, a 2.60-percentage-point improvement. These comparisons now rest on experiments conducted under our shared evaluation protocol.

Thank you for your detailed response. Most of my concerns have been addressed. I will update my score.

AuthorsOpenReview ↗

Thank You for Your Constructive Feedback

Thank you for your careful review and constructive feedback. Your suggestions have helped us strengthen the experimental evaluation and clarify our contributions. We appreciate your time and engagement throughout the discussion and will add the additional results into the final version.

AuthorsOpenReview ↗

Response to Reviewer aFEv: Thank You for Your Support and Our Further Discussion on Design Principles

Dear Reviewer aFEv,

Thanks a lot for supporting this paper based on your review and discussion. Reviewer E7MV still thought the method looks simple and may not handle long-term conversational memory well, especially with sophisticated constraints, updates, failure modes, etc. Then we presented a case study to show that using the EnSI index to retrieve truly relevant passages and feed such “short context" to LLM to explore its reasoning and generating power could be an effective way to handle many problems.

We include this discussion here to show the power of EnSI-retrieval + LLM. If you are convinced this is a smart way to handle the problem, please express your further support.

P.S.

Our answer to Reviewer E7MV.

Thank you for expressing your further concern to us. In the very last comment of yours, you wrote: “More importantly, … the paper applies entity-structured indexing to the memory scenario without extending our understanding of what such an index must do differently for long-term conversational memory—what new constraints, failure modes, or design principles it entails. This is a good comment for us to discuss in detail, that is, how to use our index to handle long-term conversational memory, especially with new constraints, failure modes, or design principles.

Our current design follows a simple philosophy: The memory system retrieves the evidence needed to answer a question, while the LLM takes such relatively “short” context to answer questions based on its own reasoning/generative power.

1. How does EnSIMem handle reasoning in long-term conversational memory?

It retrieves the most relevant cases (including constraints/updates) (treat it as small/localized context) and LLM can easily figure out the detailed inference logic”. This avoids additional complex methods (e.g., training and reinforcement learning) to take care of complex constraints, updates, or failures.

For example, we show two cases from LoCoMo dataset below.

CaseQuestionReference AnswerEnSIMemMemora (P)
1. Historical state: conv-44, q0047How many pets did Andrew have, as of September 2023?OneOne — correctTwo — incorrect
2. Event counting: conv-42, q0050How many times has Joanna's scripts been rejected?TwiceTwice — correctThree times — incorrect

In case 1, the relevant evidence is dispersed across conversations over several months: Andrew initially reports having no pets, mentions Toby (a pet) in August, then introduces the additional dog Buddy in October, and explicitly describes having two dogs in November. Answering the question about September requires bringing these separated experiences together and interpreting them with LLM.

EnSIMem treats episodes as independently retrievable units, so relevant experiences do not need to be adjacent in the conversation history. Entity–property requirements guide access to the relevant episodes, which can then be assembled into a small answering context. Retrieving an older experience does not require including all intervening conversations or traversing a sequence of neighboring chunks. Temporal distance therefore does not impose an additional chain of retrieval steps, although retrieval quality still depends on matching and evidence selection.

Once these dated experiences are available, the LLM can perform the inference: Toby is present before September, while Buddy appears as a later addition in the October conversation. EnSIMem therefore answers one. The memory system supplies the dispersed evidence; the LLM applies the temporal constraint.

Case 2 can be reasoned similarly (omitted due to word limit).

This illustrates an important principle for long-term memory: the organization used to access an experience should not depend on its position in the conversational sequence. Independently accessible episodes allow temporally distant evidence to become locally available for reasoning.

Memora (P) also retrieves relevant dated information in this example, but answers two. Its error concerns the interpretation of changing states: the later two-dog state is treated as applicable to September (but happened in October). Only retaining very detailed original information can answer it correctly (not the summary abstracts, as done by Memora).

2. EnSIMem currently does not handle sophisticated reasoning and updates via long chains.

This is our current research focus, by adding additional update/reasoning structures to entity-centered index structure (i.e., extending properties/relations to additional class of attributes handling conditions, updates, and reasoning). This has been mentioned in our rebuttal to reviewer aFEv. We believe it is an interesting enhancement to EnSIMem but it is beyond the scope of this paper. One cannot cover all the potential extensions in one paper.

Reviewer_3k99OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EnSImem, an entity-structure-indexed long-term memory architecture that separates offline memory construction from query-time retrieval. It partitions dialogues into theme-coherent episodes, indexes dialogue-grounded entity–property records linked to source turns, and uses requirement-aware planning with structured matching and dense fallback to build evidence-grounded contexts. Experiments on long-term conversational memory benchmarks support the design, with ablations validating the episode and property granularity choices.

Critical strengths

1.Sound design and well motivation: The entity-structure indexing-based long-term memory architecture proposed by EnSImem is reasonable for long-term memory and addresses issues such as matching ambiguity in dense retrievers.

2.The experimental evaluation is adequate: the main experiments show that EnSImem compares favorably with comparable methods, and the ablation studies validate the contribution of each component.

3.Training-free and cost-effective: The method does not rely on supervised training, lowering the barrier to practical deployment.

Critical weaknesses

1.LLM dependence and potential bias: The construction pipeline relies on LLMs and may therefore inherit LLM-specific biases. The authors are encouraged to report the pipeline’s performance when different LLMs are used for extraction, so as to verify the stability of the proposed method.

2.The paper discusses only deployment cost and does not consider extraction cost. A discussion of extraction cost would also be valuable for practical deployment.

3.Higher latency: Although the overall number of input tokens is reduced, latency increases, which may constitute a trade-off for practical online deployment. The authors are encouraged to further discuss this trade-off.

Questions for authors

see above

3 comments
AuthorsOpenReview ↗

Follow-Up Response to Reviewer 3k99: Did Our Previous Responses Address Your Concern and Any Other Questions?

Thank you again for your time and thoughtful feedback. We have provided additional experiments and responses addressing your concerns about LLM dependence, offline construction cost, and the latency trade-off.

We would appreciate hearing whether any concerns remain or whether further clarification is needed. We are happy to answer any questions you have.

If you find that our responses have addressed your concerns and you think EnSIMem is a good approach, we would be grateful if you could support it even further. Thank you for considering our work.

Thank you for your response, and I have no further questions.

AuthorsOpenReview ↗

Response to Reviewer 3k99: Thanks for Your Support and a Further Discussion on Design Principles

Dear Reviewer 3k99,

Thanks a lot for supporting this paper based on your review and discussion. Reviewer E7MV still thought the method looks simple and may not handle long-term conversational memory well, especially with sophisticated constraints, updates, failure modes, etc. Then we presented a case study to show that using the EnSI index to retrieve truly relevant passages and feed such “short context" to LLM to explore its reasoning and generating power could be an effective way to handle many problems.

We include this discussion here to show the power of EnSI-retrieval + LLM. If you are convinced this is a smart way to handle the problem, please express your further support.

P.S.

Our answer to Reviewer E7MV.

Thank you for expressing your further concern to us. In the very last comment of yours, you wrote: “More importantly, … the paper applies entity-structured indexing to the memory scenario without extending our understanding of what such an index must do differently for long-term conversational memory—what new constraints, failure modes, or design principles it entails. This is a good comment for us to discuss in detail, that is, how to use our index to handle long-term conversational memory, especially with new constraints, failure modes, or design principles.

Our current design follows a simple philosophy: The memory system retrieves the evidence needed to answer a question, while the LLM takes such relatively “short” context to answer questions based on its own reasoning/generative power.

1. How does EnSIMem handle reasoning in long-term conversational memory?

It retrieves the most relevant cases (including constraints/updates) (treat it as small/localized context) and LLM can easily figure out the detailed inference logic”. This avoids additional complex methods (e.g., training and reinforcement learning) to take care of complex constraints, updates, or failures.

For example, we show two cases from LoCoMo dataset below.

CaseQuestionReference AnswerEnSIMemMemora (P)
1. Historical state: conv-44, q0047How many pets did Andrew have, as of September 2023?OneOne — correctTwo — incorrect
2. Event counting: conv-42, q0050How many times has Joanna's scripts been rejected?TwiceTwice — correctThree times — incorrect

In case 1, the relevant evidence is dispersed across conversations over several months: Andrew initially reports having no pets, mentions Toby (a pet) in August, then introduces the additional dog Buddy in October, and explicitly describes having two dogs in November. Answering the question about September requires bringing these separated experiences together and interpreting them with LLM.

EnSIMem treats episodes as independently retrievable units, so relevant experiences do not need to be adjacent in the conversation history. Entity–property requirements guide access to the relevant episodes, which can then be assembled into a small answering context. Retrieving an older experience does not require including all intervening conversations or traversing a sequence of neighboring chunks. Temporal distance therefore does not impose an additional chain of retrieval steps, although retrieval quality still depends on matching and evidence selection.

Once these dated experiences are available, the LLM can perform the inference: Toby is present before September, while Buddy appears as a later addition in the October conversation. EnSIMem therefore answers one. The memory system supplies the dispersed evidence; the LLM applies the temporal constraint.

Case 2 can be reasoned similarly (omitted due to word limit).

This illustrates an important principle for long-term memory: the organization used to access an experience should not depend on its position in the conversational sequence. Independently accessible episodes allow temporally distant evidence to become locally available for reasoning.

Memora (P) also retrieves relevant dated information in this example, but answers two. Its error concerns the interpretation of changing states: the later two-dog state is treated as applicable to September (but happened in October). Only retaining very detailed original information can answer it correctly (not the summary abstracts, as done by Memora).

2. EnSIMem currently does not handle sophisticated reasoning and updates via long chains.

This is our current research focus, by adding additional update/reasoning structures to entity-centered index structure (i.e., extending properties/relations to additional class of attributes handling conditions, updates, and reasoning). This has been mentioned in our rebuttal to reviewer aFEv. We believe it is an interesting enhancement to EnSIMem but it is beyond the scope of this paper. One cannot cover all the potential extensions in one paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All Reviewers: A Conceptual Level Discussion about EnSIMem's Novelty and Efficiency

On the innovation and efficiency of the EnSIMem approach.

Thanks to all the reviewers for your time and effort at careful reviewing and providing thoughtful comments. It seems the innovation and efficiency of our approach should be further clarified to all reviewers. We further discuss these points at the general (i.e., conceptual) level (also supported by our experiments) as follows.

G1. Technical novelty in comparison with the previous approaches.

Previous approaches either explore good and sufficient training data with reinforcement/other learning methods (thus abbreviated as “training”) or build summarized graph structures to handle agent memory/actions (thus abbreviated as “graph”). However, for training, one may not always get sufficient and well-fit training data to make it perfect. For graphs, it could be difficult to handle (i) subtle tasks (since the graph is freezed at the construction time), (ii) long and diverse corpus (since one has to find and link with remote entities), and (iii) dynamic updates (one may need to search for potentially remote connections and update the graph dynamically). Our approach does not need training, naturally handles long documents and dynamic updates, adapts to subtle tasks, and preserves the original information. Thus, this is a fundamentally different and innovative approach. It may change the landscape of the current agent memory design and we hope the reviewers can see our points. Please feel free to disagree with us and raise any further questions on this issue.

G2. On the cost/efficiency of on-line, offline and incremental updates.

On construction and search efficiency: The major cost/efficiency difference between EnSIMem and a graph-based one is that the graph-based approach needs to construct graphs at memory construction time, but ours needs to only construct local EnSI-index. Thus construction is efficient for ENSIMem. However, when an online task comes, a graph-based approach, relying on a fixed graph (links freezed at the construction time), cannot adapt subtle tasks which may require different links based on the online semantics. Ours leaves this resolution at the query/task execution time and will choose top-k potential matching points based on different queries/tasks. Since ours checks multiple potential matching points dynamically to adapt to different tasks, it leads to better quality but takes a little more time. This approach trades flexibility/quality against speed (not blindly following “frozen links”). A discussion on potential further improvements are stated in the paper but we believe this is beyond the scope of this paper.

On incremental update efficiency: When new sessions arrive, EnSIMem partitions new sessions into episodes, extracts structured records as new indexes, and appends them (together with the links to the new passages). There is no change to earlier episodes and indexes. This light incremental update is a major advantage of EnSIMem. For a graph-based approach, a new session means you will need to reconstruct the previously established graph, which is costly since you may need to search for potential new connections with quite distant records. Ours is completely localized. Only at query time, the truly task-matched index needs to be fetched, which takes minimal update cost.

Area_Chair_XwVSOpenReview ↗

Discussion Window Closing in 2 days. Please Respond to the Author's Rebuttal!

Dear Reviewers,

The authors have thoughtfully responded to your reviews and the discussion window is coming to a close on November 18th. Please take a minute to engage with their response and discuss whether it changes your opinions.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.