acceptodds
Under review as a conference paper at ICLR 2027

One Cue, the Right Episode: Evidence Recovery for Dialogue Memory

Abstract

With ever-longer interaction histories, LLM-based conversational systems face a central challenge: how can they retrieve the most discriminative evidence for a response within a limited reader-context budget? The human brain tackles this via hippocampal separation, completion from partial cues, and selective readout. Inspired by this tripartite organization, we propose Sparse Event Addressing Memory (SEA-Mem). SEA-Mem first encodes each turn into immutable traces, source atoms, and sparse bindings. When a query arrives, a planner groups its information needs, and then a simple three-step mechanism resolves the memory access: DG-Address pinpoints the episode via sparse addressing, CA3-Complete recovers evidence through binding-following, and CA1-Select selects a budgeted set of turns that jointly supports the query. Together, they let associations guide access, while the selected original turns—not the associations—remain the evidence presented to the answerer. With lightweight association construction and compact readout, SEA-Mem achieves strong answer quality on LongMemEval and LoCoMo under a fixed source budget. On a matched LongMemEval subset, accuracy is largely preserved from Oracle evidence to full histories. Our experiments further show that SEA-Mem is more resistant to semantically confusable distractors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7q4ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SEA-Mem for recovering complementary evidence from long dialogue histories under a limited source budget. It combines query-guided source addressing, one-hop associative completion, and LLM-based source selection while retaining original turns as answer evidence. Experiments on LongMemEval and LoCoMo examine answer quality, interference robustness, efficiency, and component contributions.

Critical strengths
  1. The paper addresses a meaningful problem of retaining complementary evidence when semantically similar memories compete for limited context. Distinguishing candidate evidence recovery from final evidence retention provides a useful perspective on dialogue memory.
  2. SEA-Mem maintains clear source provenance throughout retrieval and selection. Using associations to guide access while presenting original turns to the answerer makes the evidence pathway traceable.
  3. The evaluation covers two benchmarks and two backbones, with controlled interference experiments and component ablations. Freezing memory construction across interference conditions helps examine retrieval behavior separately from writing-stage changes.
Critical weaknesses
  1. The motivation for retaining complementary evidence under a limited source budget is clear, but SEA-Mem’s technical advance over existing associative memory methods remains insufficiently established. The paper should identify its distinctive design and explain which limitation of prior methods it addresses.
  2. In Section 4.2, SEA-Mem uses CA1-Select, whereas Dense-native and AssoMem-native directly return their top-k sources. Although the component ablations are informative, the advantage over these baselines with the same selector remains unclear.
  3. The aggregate and category-level results in Tables 1 and 4 are inconsistent. These discrepancies make the reported performance and robustness to history expansion difficult to verify.
  4. The interference experiments provide useful evidence, but the accuracy–token trade-off remains unclear. Section 4.2 varies the number of sources rather than the token budget, while Table 7 reports token usage separately, leaving it unclear how much context SEA-Mem can save without sacrificing accuracy and when its advantage diminishes as more context is allowed.
Questions for authors
  1. What is SEA-Mem’s principal technical distinction from the associative-memory approaches discussed in the paper, such as AssoMem, HippoRAG-2, and A-MEM? Please clarify which limitation these approaches have that SEA-Mem addresses, and what evidence demonstrates the benefit of its distinctive design. (refer to Weakness 1)
  2. Does SEA-Mem retain its advantage when Dense and AssoMem candidates are processed by the same CA1 selector, with comparable candidate budgets and the same final source limit? (refer to Weakness 2)
  3. Can you reconcile the aggregate and category-level results in Tables 1 and 4? For example, weighting SEA-Mem’s DeepSeek Oracle category scores by the standard category counts gives approximately 82.1%, whereas Table 1(a) reports 79.4%. (refer to weakness 3)
  4. Could you compare SEA-Mem and the baselines across matched Reader token budgets, including larger budgets, and report accuracy, distractor occupancy, and total query-time LLM tokens on the same examples? LLMs are capable of processing large numbers of tokens. Reducing the input from, for example, 8K to 4K tokens alone does not demonstrate that the retained memories better support answering. Please clarify what information SEA-Mem removes and whether its benefit comes from eliminating confusable or irrelevant memories while preserving necessary evidence, or from another mechanism.
4 comments
AuthorsOpenReview ↗

To Reviewer 7q4Z — Part 1/2

We are grateful for your time and effort in reviewing our work. To address your questions, we clarified SEA-Mem's distinction from prior methods, tested retrieval with a shared CA1 selector, corrected score discrepancies, and compared accuracy, distractor occupancy, and token costs across matched Reader budgets.

Q1. Distinction compared to other methods

SEA-Mem's key distinction is combining event discrimination with evidence recovery: it isolates the target episode using query-specific cues, follows associations to recover complementary details, and selects a compact set of original turns that jointly support the answer.

MethodLimitationSEA-Mem designEvidence
AssoMemIndividual utterance ranking may misassign high scores to turns from a different event simply because they share the same entity and action as the question, letting distractor turns from similar sessions occupy a Reader slot.To avoid Reader slots being taken by similar distractor sessions, DG-Address first locates target-event turns using entity and action cues. To filter out these distractors, CA1-Select then jointly evaluates the candidate pool and retains only answer-supporting turnsDespite comparable evidence recall (78.0% vs. 78.6%), SEA-Mem admitted fewer similar-session distractors into the Reader context (5.1% vs. 17.3% occupancy) and achieved higher judged accuracy (76.0% vs. 60.7%) under Semantic-50 interference at k=5 (Table 2).
HippoRAG-2Graph propagation ranks passages by connectivity, so top passages may repeat one detail while omitting another required detail.To recover missing details, CA3-Complete follows DG-Address links to retrieve complementary source turns. To retain them within the readout budget, CA1-Select jointly evaluates these candidates to select up to K turns that collectively support the answer.For the 281 LoCoMo multi-hop questions (analysis above Table 5), SEA-Mem retrieved at least one gold turn on 75.8% of questions, compared with 64.8% for HippoRAG-2. Consequently, HippoRAG-2 completely missed all annotated evidence on 99 questions, compared to only 68 for SEA-Mem. These misses illustrate how HippoRAG-2 can omit complementary evidence needed to answer multi-hop questions.
A-MEMQuery-driven retrieval followed by write-time link expansion can let topically related turns fill the Reader’s context while omitting the turn that answers the question.To reduce distractors, DG-Address prioritizes entity/action matches, and CA1-Select retains only jointly covering turns. To recover evidence, the Writer links each fact to its source turn, letting DG-Address access it directly and CA3-Complete add complementary details via links.A-MEM passed 35× as many history tokens to the Reader on LongMemEval-S60, while achieving lower LoCoMo accuracy than SEA-Mem (47.2% vs. 58.5%). This suggests that link expansion can bring in context that contributes little to the answer.

Q2. Matched Budgets & Shared CA1 Selector

To isolate whether SEA-Mem's advantage stems from its selector or its retrieval mechanism, we use the same LongMemEval-S questions. Dense and AssoMem match SEA-Mem’s candidate count using their top-ranked turns. We compare Native readout (top 5 turns directly to the Reader) against Shared CA1 (applying SEA-Mem’s CA1 selector to each candidate pool to select up to 5 turns). Results:

MethodNative rankingShared CA1Gain over native
Dense60.7%68.0%+7.3 pp
AssoMem67.3%71.3%+4.0 pp
SEA-Mem—78.7%—

Adding CA1-Select can improve accuracy in other method, however, DG-Address and CA3-Complete jointly improve accuracy beyond CA1-Select alone.

AuthorsOpenReview ↗

To Reviewer 7q4Z — Part 2/2

W3&Q3. Experimental data reconcile the aggregate

Thank you for bringing this discrepancy to our attention. We identified an error in the original aggregation and recomputed the scores from the question-level results using the evaluation protocol described in the manuscript.

  • Table 1. SEA-Mem’s overall DeepSeek accuracy on LongMemEval-Oracle remains 79.4% (397/500). The corrected category scores in Table 1(b), in column order, are 64/78 (82.1%), 67/70 (95.7%), 23/30 (76.7%), 50/56 (89.3%), 105/133 (78.9%), and 88/133 (66.2%). These counts sum to 397 correct answers across 500 questions, matching Table 1(a).

  • Table 5 (Table 4 in the original submission). The only error in this table was HippoRAG-2’s overall F1. We corrected it from 40.0% to 38.6% in both Tables 1(a) and 5. All category-level F1 scores were correct.

Q4. Accuracy-Efficiency Trade-off across Matched Token Budgets

To test if SEA-Mem’s advantage persists across growing Reader budgets, we compared SEA-Mem, AssoMem, and Dense on same LongMemEval-S questions (Semantic-50 interference). We applied four matched evidence caps (256 to 4,096 tokens) while keeping all candidate pools fixed (SEA-Mem retained its full native pool); only CA1 was adapted, replacing the five-turn limit with a token cap to select a sufficient, nonredundant set without forced filling or designated core turns. Keeping native rankings for baselines, all methods used identical packing rules, Reader, and Judge to report accuracy, distractor occupancy, and estimated query-time LLM tokens.

Reader evidence capMethodJudged accuracy ↑Distractor occupancy ↓Query-time LLM tokens/question ↓
256SEA-Mem72.7%1.5%1,364
AssoMem56.7%25.2%328
Dense49.3%32.7%327
512SEA-Mem77.3%2.2%1,511
AssoMem66.0%31.0%525
Dense62.7%35.3%523
1,024SEA-Mem80.0%2.2%1,776
AssoMem71.3%32.3%915
Dense69.3%37.6%911
4,096SEA-Mem80.0%2.3%1,778
AssoMem78.7%52.6%3,228
Dense77.3%61.7%3,170

Across all 150 Semantic-50 questions, SEA-Mem kept distractor occupancy low even at the 4,096-token budget: 2.3% for SEA-Mem, compared with 52.6% for AssoMem and 61.7% for Dense. We found that SEA-Mem primarily filters out topically similar memories about different entities or occurrences. For example, it excluded advice about the Fitbit Inspire HR while retaining the user’s Charge 3 updates. It also excluded a weekend trip with friends while retaining the account of the user’s week-long family trip to Hawaii.

Thank you for your rebuttal. My main concerns are addressed, and errors are corrected; thus, I update my rating to 3: weak acceptance.

Thank you for your careful review and for raising your rating. Your feedback helped us clarify the technical distinctions, strengthen the controlled comparisons, and improve the accuracy of our reported results.

Reviewer_iNvLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses evidence retrieval for long-term dialogue memory, focusing on how to pinpoint the right event and retain complementary details under a tight evidence budget when the history contains many similar distractors. The authors propose SEA-Mem, which extracts source-linked structured atoms during writing and connects them across turns via deterministic keys. At retrieval time, it groups evidence requirements, locates initial sources, performs one-hop associative completion, and jointly selects original dialogue turns within a token budget. The method shows strong results on LongMemEval and LoCoMo, backed by controlled interference studies, budget analyses, and ablations.

Critical strengths
  1. The paper effectively distinguishes retrieving relevant content from retaining sufficient evidence within a limited budget. The division into addressing, completion, and joint selection is practically meaningful: initial cues may retrieve only partial evidence, associative expansion recovers complementary information, and final selection controls redundancy.
  2. In Table 1, SEA-Mem achieves 78.0% accuracy on LongMemEval-S with the DeepSeek backbone, compared with 69.2% for the strongest baseline, and retains its advantage with Qwen.
Critical weaknesses
  1. It is not entirely clear how much SEA-Mem gains over a simpler retrieve-then-select pipeline. In the interference experiments, Dense and AssoMem rely on their native rankings, whereas SEA-Mem adds query planning and LLM-based joint selection. While the DG ablation shows structured addressing helps, it also drops DG-dependent partial-source recovery. Thus, we cannot tell how close a standard retriever would get if it simply fetched a larger candidate pool and fed it into the same CA1 selector.
  2. Event association keys require exact matches between normalized entities, relations, and arguments. This may miss different descriptions of the same event while connecting repeated events occurring at different times. State association keys further omit attribute values, linking multiple updates to the same attribute. The existing experiments with semantically similar distractor sessions do not sufficiently establish whether this design remains effective when similar events repeatedly involve the same entity.
  3. Because DG admits all verified sources and one-hop expansion is uncapped, CA1 may have to scan a very large candidate pool. In Table 7, SEA-Mem keeps the final answering context tiny (175 tokens), but consumes ~89k online tokens and ~220k lifecycle tokens that both higher than Dense. Without candidate pool statistics and a cost breakdown across planning and selection, it is hard to judge how well this scales as dialogue histories grow.
Questions for authors
  1. If Dense or AssoMem retrieved a similarly sized candidate pool and used the same query plan and CA1 selector, how much of SEA-Mem’s margin would remain?
  2. When someone repeats similar activities or an attribute is updated multiple times, where does temporal filtering happen? Could you show an example of how the system avoids merging details from separate occurrences into a single event?
  3. What are the median and tail sizes of the candidate pool inspected by CA1, and how does the online token cost break down across planning, semantic supplementation, and CA1 selection?
1 comment
AuthorsOpenReview ↗

To Reviewer iNvL

Thank you very much for your careful reading and insightful comments. In response to your questions, we compared retrievers using the same selector, tested how CA1 distinguishes repeated updates by date, and measured candidate pool sizes and query-time token costs.

W1&Q1. True gain of SEA-Mem & isolating the confounded partial-source recovery effect in the DG ablation

To assess how much of SEA-Mem’s advantage remains when planning and source selection are held fixed, we matched Dense and AssoMem to SEA-Mem’s candidate count for each question. Each retriever supplied its own candidates, and the same CA1 selector used the same frozen query plan to choose up to five source turns for the Reader. The results are:

MethodNative rankingShared CA1Gain from CA1
Dense60.7%68.0%+7.3 pp
AssoMem67.3%71.3%+4.0 pp
SEA-Mem—78.7%—

With candidate counts matched per question and the same frozen query plan and CA1 selector, SEA-Mem retained a 10.7-percentage-point accuracy advantage over Dense and a 7.4-point advantage over AssoMem.

W2&Q2. Repeated similar activities and temporal filtering

For W2, in SEA-Mem we designed several steps to eliminate the interference from similar events that repeatedly involve the same entity. When the same entity receives several updates to the same attribute, the shared key links those updates for retrieval. The Writer keeps each update tied to its original turn, preserving which value applied on which date. When several updates reach CA1-Select, it uses the date requested in the question to select the turn that supports the corresponding value.

For Q2, SEA-Mem preserves temporal information at write time and explicitly applies it during source selection.

  • Write — Preserve wording: The Writer links each fact to its original turn, retaining the exact dates or relative expressions.
  • Plan — Extract intent: The query plan makes temporal constraints explicit (e.g., "my most recent visit" → selection=last), guiding CA1-Select.
  • Retrieve — Gather occurrences: DG-Address locates events via entity and action cues; CA3-Complete follows links to gather related turns, presenting different visits as distinct candidates.
  • Select — Filter temporally: CA1-Select matches candidates against the query plan.

Example: repeated visits to the same market. LongMemEval-S question 7e974930 asks: “How much did I earn at the Downtown Farmers Market on my most recent visit?”

Original user turnFacts stored by the WriterRetrieved by SEA-Mem?Passed to the Reader?
s14:t2 — lists two visitsTwo distinct facts from the same turn: March 18: $220; May 6: $350Yes — one candidate turnNo
s44:t6 — identifies the most recent visit$420 at the same market, explicitly described as “on my most recent visit”Yes — one candidate turnYes — the only turn selected

Both sales records reached CA1-Select. Guided by the query plan’s "most recent" requirement, CA1 excluded the earlier sales (s14:t2) and passed only the latest turn (s44:t6, stating $420) to the Reader, which answered $420.

W3&Q3. Median and tail sizes of the CA1‘s candidate pool & online token breakdown

For Q3, on the 60 LongMemEval-S questions used for efficiency evaluation, CA1 inspected a median of 5 candidate source turns per question (95th percentile: 17; maximum: 25). The distribution was right-skewed: 63.3% had exactly 5 candidates, 28.3% had 6–15, and 8.3% had 16 or more.

As for Online token cost. The full breakdown is:

Online stage or componentLLM inputLLM outputShared Embedding input
Query planning372.236.00
Embeddings0084,070.7
DG-Address003,411.3
CA1-Select789.96.20
Reader context175.0—0
Other Reader prompt tokens197.3—0
Answer generation (output)—30.40
Online total1,534.472.687,481.9

The table reports the embedding inputs for the initial query over each history. Since embeddings of historical turns are cached and reused, subsequent queries only require fresh encoding for unseen inputs, such as a new question or an added turn.

Reviewer_5rqVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SEA-Mem, which extracts facts from dialogue history, links related turns, and selects a few original turns as evidence for answering. Evaluation on LongMemEval and LoCoMo with two LLM backbones show improved accuracy and robustness to distracting memories under a limited readout budget.

Critical strengths
  1. Using original turns as evidence preserves details that extracted facts may miss and makes the sources easy to check. Accuracy gains hold across both backbones and benchmarks.

  2. The interference study fixes stored memories and matches distractor volume. Evidence recall before and after selection helps locate failures.

Critical weaknesses
  1. How much the access rules add beyond planning and LLM selection remains unclear. Baselines use native rankings rather than the same selector. The count-matched DG ablation's 24.7-point drop also includes removing partial-source recovery (65/150 questions; Appendix B.4). A Dense control with the same planner and selector would help. So would a simple agent searching raw history with the same model and a similar query-time budget.

  2. DG and CA3 leave the candidate pool uncapped (see Appendix A.3). Since association keys omit time, repeated events or updates can increase CA1's workload even when k is small. Appendix B.3 combines LLM and embedding tokens, so the 175-token Reader context does not establish low overall query cost?

Questions for authors
  1. How much of the DG ablation's accuracy drop remains if partial-source recovery is retained?

  2. What are CA1's mean and 95th-percentile candidate counts? Could Table 7 separate its LLM token cost from embedding tokens?

  3. Were Table 1(a) and (b) computed from different runs or subsets? Weighting panel (b)'s DeepSeek SEA-Mem Oracle scores by the official category counts gives about 82.1%, not 79.4%.

3 comments
AuthorsOpenReview ↗

To Reviewer 5rqV

We sincerely thank you for your time, efforts, and careful evaluation of our manuscript. Based on your suggestions, we added controls to measure the contribution of the partial source, compared retrievers under the same query plan and CA1-Select, and reported CA1 candidate counts plus separate LLM/embedding token costs. We also rechecked all 500 Oracle judgments and corrected the category-level scores in Table 1(b). We address each question below.

W1&Q1. Missing dense control with the same planner and selector.& Ablation of DG and partial-source recovery

For W1, we conducted an ablation study. On the same Full-S questions, Dense and AssoMem each supplied their highest-ranked turns up to SEA-Mem’s candidate count for that question. Under a shared readout setting, the same CA1 selector used the same frozen query plan to choose up to five turns from each candidate pool. The results show that the shared planner + CA1 selection mechanism alone yields a performance improvement, but this gain is maximized when combined with DG/CA3.

RetrieverNative accuracy (%)Shared CA1 accuracy (%)Shared CA1 evidence recall (%)
Dense60.768.069.1
AssoMem67.371.372.9
SEA-Mem—78.780.3

For Q1, we report the four manuscript settings (A–D) and an additional paired control (E) on the same 150 LongMemEval-S questions. In E, we replace the native DG entrances with the same number of Dense entrances per evidence group, recompute CA3 from those entrances, and retain the same partial source as the native-DG control whenever available.

SettingJudged accuracy (n = 150)
A. Native DG + partial (Full)78.7% (118)
B. Dense, partial off54.0% (81)
C. Partial source only6.0% (9)
D. Native DG, partial off76.0% (114)
E. Dense + partial56.0% (84)

The paired controls confirm that replacing DG entrances causes the primary performance gap, not the removal of partial-source recovery. Thus, the original w/o DG gap is predominantly driven by the absence of DG entrances, with partial-source recovery serving only as a secondary, DG-dependent mechanism.

Q2. CA1’s mean candidate count & Report LLM and embedding token costs separately

We measured CA1 candidate counts on the 60 LongMemEval-S questions used for efficiency evaluation.

Mean distinct candidates inspected by CA195th percentileMaximum source turns passed to the Reader
7.33175

The candidate counts were right-skewed: CA1 inspected 5 or less candidates on 63.3% of questions , 6–15 on 28.3%, and 16 or more on the remaining 8.3%. This upper tail pulled the mean to 7.33, above the median of 5.

As for tokens, we report mean tokens per question across the full lifecycle, LLM tokens include model inputs and outputs while embedding tokens count model inputs.

MethodLLM tokensEmbedding tokens
Dense850.484,035.1
AssoMem85,595.583,274.0
SEA-Mem132,033.287,481.9
HippoRAG-2389,313.9104,188.8
REMem315,512.3244,187.6
A-MEM2,201,143.7496,577.4
Mem01,698,790.766,142.1

Q3. Numerical inconsistency

Thank you for carefully checking the category scores and bringing this discrepancy to our attention. We apologize for the arithmetic error in Table 1(b); the results are derived from the exact same subsets described in the manuscript. To verify the DeepSeek SEA-Mem Oracle result, we returned to the original 500 question-level judgments and recomputed accuracy in two ways: directly over all questions and separately within each of the six categories. Both calculations use the same predictions and judgments from a single run.

The overall count is 397/500 (79.4%), as reported in Table 1(a). In the order shown in Table 1(b), the category counts are 64/78, 67/70, 23/30, 50/56, 105/133, and 88/133. The numerators sum to 397 and the denominators to 500, so the category-weighted accuracy is also 79.4%. We have corrected the Oracle entries in Table 1(b) to 82.1%, 95.7%, 76.7%, 89.3%, 78.9%, and 66.2%, respectively.

Thanks for the extra experiments and corrections. The new controls address my ablation concern, and Table 1 now adds up. The cost breakdown is clear too. I'll keep my rating at 3.

Thank you for your careful review and constructive feedback, which helped us improve the experiments and clarify the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Update Notes for Paper Draft

The revised manuscript includes the following updates.

  • Sections 1–2 clarify how SEA-Mem differs from other associative-memory methods. (7q4Z)
  • Tables 1 and 5 correct SEA-Mem’s LongMemEval-Oracle category scores and HippoRAG-2’s LoCoMo F1. (5rqV, 7q4Z)
  • Section 4.4 and Appendix B.4 add DG ablations and compare retrievers using the same query plan and CA1 selector. (5rqV, iNvL, 7q4Z)
  • Appendix B.2.1 reports accuracy, distractor occupancy, and query-time LLM tokens under matched Reader token budgets. (7q4Z)
  • Appendix B.2.2 tests whether CA1 selects the update corresponding to the requested date. (iNvL)
  • Appendix B.3 reports CA1 candidate counts and separates LLM tokens from embedding tokens. (5rqV, iNvL)

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. We hope that our revisions and responses have adequately addressed their questions and concerns, and we look forward to further discussions with the reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.