- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EnSIMem is a training-free memory pipeline for long-term conversational agents. Offline, an LLM splits each session into contiguous, theme-coherent episodes and extracts [entity][type][property:value] records, each linked to its source turns. Online, an LLM planner turns the query into evidence requirements and assigns a reasoning type. Retrieval combines structured entity–property matching with dense fallback and stops once the requirements are covered. GPT-4.1-mini then answers from the expanded original episodes, so the records serve as addresses rather than as evidence. The paper reports 90.6% on LoCoMo (MEMORA(P): 86.3%) and 92.8% on LongMemEval (87.4%). It adds subset ablations on episode granularity, property granularity, structured vs. dense retrieval, and adaptive vs. fixed top-k. Online latency is 14.4s, against 5.70s for MEMORA.
Critical strengths
Strong, protocol-checked headline results. On LoCoMo the method gains +4.3pp overall over MEMORA(P), including +11.4pp on open-domain and +10.0pp on multi-hop. It also beats Full Context (82.5%) while using fewer context tokens (7,470). The authors re-ran MEMORA(P) locally and obtained 86.23% against the published 86.3% (Sec. 4.1), which supports comparability with that baseline. The two judges agree closely (App. D, r > 0.93). With 1,540 questions, the main gap is well above sampling noise.
Inspectable, training-free design. Separating localization (records) from interpretation (source episodes) produces traceable answers (App. B). The design is a reasonable principle for practical memory systems.
Critical weaknesses
The central mechanisms are not causally isolated. All ablations use an undisclosed "fixed LoCoMo subset." The reported accuracies (96.59 / 94.32 / 90.91 / 86.36%) correspond to 85 / 83 / 80 / 76 out of 88 questions. The granularity effects therefore rest on 2–5 questions, and the adaptive budget beats fixed top-8 by one question (Fig. 3). The subset is also clearly easier than the full set (96.6% vs. 90.6%). More importantly, the two claims the paper rests on are never ablated: (a) answering from expanded episodes versus from records alone (Sec. 3.1, "address, not substitute"), and (b) the requirement-aware planner, which accounts for 10.3s of the 14.4s latency. It is also unclear whether the dense-only variant in Table 4 keeps the planner and the adaptive budget. Finally, the extraction prompt (App. E.1) hard-codes the predicates "research, attend, read, paint, support," which match the conv-26 case studies, and there is no development/test split on a 10-conversation benchmark.
The headline comparisons are not controlled, and Table 2 is inconsistent. All baseline rows are transcribed from Memora, and the paper does not state that they share the GPT-4.1-mini answer model or the same question set. On LongMemEval, every transcribed row reconciles exactly with the LongMemEval-S category sizes (30/56/133/133/78/70) and their weighted average. EnSIMem's rows do not: 91.7% (SS-assistant), 89.3% / 89.1% (temporal), 93.0% / 91.4% (multi-session), and 96.1% (knowledge update) are unattainable fractions at those sizes. The category-weighted average is 92.6%, not the reported 92.8%. App. D's max |Δ| = 3.37pp on SS-preference is also not a multiple of 1/30. This points to a different question set or a different aggregation, so the +5.4pp claim cannot be verified as a like-for-like comparison.
Questions for authors
Which LongMemEval variant did you use, and how many questions per category? How is "Average" computed? Please reconcile Table 2 with the standard category sizes, including how abstention items are handled.
How large is the ablation subset, how was it selected, and was it used during prompt development? Can you rerun the ablations on full LoCoMo and add two variants: (a) answering from records only, and (b) no planner, retrieving with the raw query at a matched budget?
Were the transcribed baselines run with GPT-4.1-mini as the answer model? Can you re-run at least Full Context and one or two memory baselines inside your own pipeline?
What is the offline construction cost (LLM calls, tokens, wall-clock time), especially for LongMemEval's per-question haystacks? How is the index updated when new sessions arrive?
10 comments
Response to Reviewer E7MV: New Ablations and Prompt Development
Thanks for your detailed review.
Our responses are structured into five sections:
- Previous Ablation Design and Experimental Coverage
- New Ablations and Prompt Development
- LongMemEval-S denominators and aggregation
- Controlled Baseline Comparisons
- Offline Construction and Incremental Updates
This is Section Two: New Ablations and Prompt Development.
Thank you for your comments about doing more ablations.
W1 + Q2. General comments on new ablations and prompt development
We added records-only answering and single-key retrieval without multi-hop planning on all 1,540 Category 1–4 questions across the 10 LoCoMo conversations. These experiments examine the roles of source-episode evidence and planner-guided retrieval. We also evaluated dense-only candidate retrieval on the 347 questions from conv-48 and conv-49, retaining all the other components to examine the contribution of our EnSI index. Shared components use the same experimental settings as the main evaluation. Finally, we removed the specified prompt examples, rebuilt the index, and reran the full LoCoMo evaluation. Below, we specify each intervention and report its results.
W1.4. “The central mechanisms are not causally isolated.” “More importantly, the two claims the paper rests on are never ablated: (a) answering from expanded episodes versus from records alone (Sec. 3.1, "address, not substitute"), and (b) the requirement-aware planner, which accounts for 10.3s of the 14.4s latency.” (Q2.2. “Can you rerun the ablations on full LoCoMo and add two variants: (a) answering from records only, and (b) no planner, retrieving with the raw query at a matched budget?”)
We completed the following experiments on all 1,540 Category 1–4 questions, and we report their scores now.
| Condition | Correct / Total | Accuracy |
|---|---|---|
| Full EnSIMem | 1361/1540 | 88.38% |
| Records-only answering | 1284/1540 | 83.38% |
| No planner retrieval | 191/1540 | 12.40% |
For records-only answering, we reused the source run’s selected episode identities and their order. We supplied the generation with the extracted records associated with those episodes, excluding original episode text. We retained all the other steps. This directly tests the information available to the generation when source episodes are replaced by their extracted records.
For No planner retrieval, we replaced multi-hop planning with one LLM-extracted structured key, retained all the other steps. This variant directly tests the importance of our planner.
The low score of the variant without planner indicates the importance of planner-guided evidence retrieval. EnSIMem’s planner decomposes a question into explicit evidence requirements and coordinates retrieval across intermediate entities and multiple sources. Removing this process substantially weakens the system’s ability to assemble the evidence needed for answering. These results support the value of requirement-aware planning and its associated retrieval mechanisms.
W1.5. “It is also unclear whether the dense-only variant in Table 4 keeps the planner and the adaptive budget.”
The experiment is specifically a dense-only candidate-retrieval ablation. We did not use the EnSI index in our retrieval. This examines the benefit of our EnSI index. We calculate the embedding of the whole episode and match it with the embedding of the decomposed sub-questions from the planner.
It achieved 276/347 correct, or 79.54%, compared with the full EnSIMem reference of 307/347, or 88.47%.
This experiment retains the planner and all other components and features. It shows the importance of using the EnSI index.
W1.6. “Finally, the extraction prompt (App. E.1) hard-codes the predicates "research, attend, read, paint, support," which match the conv-26 case studies, and there is no development/test split on a 10-conversation benchmark.”
We removed the benchmark-matching examples from the extraction prompt, rebuilt the memory index, and reran EnSIMem across all ten conversations. All the experiments are using the same setting, the same LLM backbone (GPT-4.1-mini), and on the same 1540 questions from LoCoMo. We also rerun Memora (S) and Memora (P) for reference. We now report the scores.
| Method | Correct / Total | Accuracy |
|---|---|---|
| EnSIMem | 1361/1540 | 88.38% |
| Memora (S), local rerun | 1320/1540 | 85.71% |
| Memora (P), local rerun | 1321/1540 | 85.78% |
The revised-prompt result remains above both Memora reruns. This supports the performance of the revised pipeline without relying on the criticized examples.
Prompt revision and development/test separation are distinct issues. We confirm that the 347-query ablation subset was not used for prompt development.
Response to Reviewer E7MV: LongMemEval-S denominators and aggregation
Thank you for your careful and detailed review.
Our responses are structured into five sections:
- Previous Ablation Design and Experimental Coverage
- New Ablations and Prompt Development
- LongMemEval-S denominators and aggregation
- Controlled Baseline Comparisons
- Offline Construction and Incremental Updates
This is Section Three: LongMemEval-S denominators and aggregation.
Thank you for checking the reported numbers carefully.
W2 + Q1. General comments on LongMemEval-S denominators and aggregation.
We now report the reran evaluation on all 500 LongMemEval-S questions and report exact numerators and denominators.
W2.2. “On LongMemEval, every transcribed row reconciles exactly with the LongMemEval-S category sizes (30/56/133/133/78/70) and their weighted average. EnSIMem's rows do not: 91.7% (SS-assistant), 89.3% / 89.1% (temporal), 93.0% / 91.4% (multi-session), and 96.1% (knowledge update) are unattainable fractions at those sizes.” (Q1.1. “Which LongMemEval variant did you use, and how many questions per category?”)
The evaluation uses LongMemEval-S, with all 500 questions, including abstention items. GPT-4.1-mini is the backbone LLM. The category denominators match those identified in the review. We now report the scores.
| Category | Correct | Total | Accuracy |
|---|---|---|---|
| Single-Session Preference | 25 | 30 | 83.33% |
| Single-Session Assistant | 52 | 56 | 92.86% |
| Temporal Reasoning | 117 | 133 | 87.97% |
| Multi-Session | 115 | 133 | 86.47% |
| Knowledge Update | 72 | 78 | 92.31% |
| Single-Session User | 70 | 70 | 100.00% |
| Overall | 451 | 500 | 90.20% |
Each percentage is computed from the corresponding integer counts. These are new, reconciled results.
W2.3. “The category-weighted average is 92.6%, not the reported 92.8%.” (Q1.2. “How is "Average" computed?” Q1.3. “Please reconcile Table 2 with the standard category sizes, including how abstention items are handled.”)
“Average” is micro-averaged accuracy. We sum the correct answers across categories and divide by the total number of evaluated questions.
For example, Accuracy = (25 + 52 + 117 + 115 + 72 + 70) / (30 + 56 + 133 + 133 + 78 + 70) = 451/500 = 90.20%.
Abstention questions remain in their original categories and in the 500-question denominator. They receive credit when the model correctly abstains. They are not excluded from the overall result.
Response to Reviewer E7MV: Previous Ablation Design and Experimental Coverage
Thank you for your careful review.
Our responses are structured into five sections:
This is Section One: Previous Ablation Design and Experimental Coverage
Thanks for your detailed questions/suggestions on performance study.
W1 + Q2. General comments on previous ablation design and experimental coverage.
We expanded the property-granularity and retrieval-budget experiments from 88 questions to all 347 Category 1–4 questions from conv-48 and conv-49. We now explicitly report the subset’s composition, selection criterion, and prompt-development history, together with exact correct counts for each condition. These experiments retain the same LLMs, embedding configuration, judge, and execution environment as the main evaluation. Below, we explain the subset selection and present the results for property specificity and retrieval budget.
W1.1. “All ablations use an undisclosed 'fixed LoCoMo subset.'” (Q2.1. “How large is the ablation subset, how was it selected, and was it used during prompt development?”)
The original subset contained 88 questions from conv-26. This time we expanded the test size to all 347 questions from conv-48 and conv-49. We now explicitly show the conversations, question counts, selection criterion, and development history.
We selected conv-48 and conv-49 of LoCoMo because their pooled full-EnSIMem accuracy was closest to the full-benchmark accuracy (Please find the table below for specific numbers). We included every Category 1–4 questions in both conversations, without filtering individual questions by correctness. These 347 queries were not used during prompt development.
W1.2. “The reported accuracies (96.59 / 94.32 / 90.91 / 86.36%) correspond to 85 / 83 / 80 / 76 out of 88 questions.” “The subset is also clearly easier than the full set (96.6% vs. 90.6%).”
We agree that the original subset was not big. Therefore, we use a larger subset containing 347 questions from conv-48 and conv-49 for the repeated ablations, and for the two new core variants we used all 1540 questions in testing. This time the subset is not easier, the full EnSIMem score on the full benchmark is very similar to the score on this subset (88.38% vs 88.47%).
W1.3. “The granularity effects therefore rest on 2–5 questions, and the adaptive budget beats fixed top-8 by one question (Fig. 3).”
We now report the new scores for the new testing on 347 questions from conv-48 and conv-49. Property granularity refers to the specificity of the property labels in the EnSI records and corresponding query-plan fields. The source episodes, record identities, entities, values, and supporting evidence are retained. This experiment does not change episode boundaries.
The difference between the default EnSIMem setting (Broad) and Broad+ is four questions out of 347. The gap increases to 12 for Very broad and 19 for Fine. Note that Broad+ means the granularity is slightly broader than the default setting.
For the retrieval budget, we evaluated six fixed numbers of episodes. Each condition uses a prefix of the same ranked candidate list.
Within this sweep, increasing the cap from top-1 to top-20 yields 95 additional correct answers. The gap between top-8 and our full EnSIMem is 17 questions.