acceptodds
Desk rejected at ICLR 2027

Hippocampus–Neocortex-Inspired Sparse-Dense Memory Self-Organization for LLM Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory is essential for LLM agents to retain and reuse information, but reliable associative access remains challenging over extended interactions. Relevant evidence may span temporally distant interactions, yet existing methods often decouple associative memory organization from fine-grained evidence retrieval and therefore miss related information. Inspired by hippocampus–neocortex interactions in human memory, we propose , a sparse–dense memory self-organization framework for LLM agents. Within the resulting memory bank, Sparse Attractor (SA) performs associative indexing over topic basins, while Dense Representation (DR) supports fine-grained memory consolidation over candidate chunks. Sparse-to-dense activation and dense-to-sparse calibration close the bidirectional interaction loop, forming a brain-inspired mechanism for memory organization. Experiments on LongMemEval, 2WikiMultiHopQA, and MuSiQue show that Lumi-Mem achieves the best R@5 on all three benchmarks and the best average QA performance with GPT-4.1-mini. Ablations with different retrievers show that SA, DR, and the sparse–dense interaction modules each contribute to performance, supporting self-organization as a practical mechanism for memory retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Long-term Memory, Sparse–Dense Interaction, Memory Organization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ka21OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Lumi-Mem, a hippocampus–neocortex-inspired memory framework for LLM agents that aims to improve retrieval of relevant information across long-term interactions. In detail, this method combines a Sparse Attractor (SA) module for associative topic-level indexing with a Dense Representation (DR) module for fine-grained evidence retrieval. Bidirectional sparse–dense interactions let topic do activation, evidence-based calibration, and adaptive fusion of retrieval scores. Experiments on LongMemEval, 2WikiMultiHopQA, and MuSiQue demonstrate improvements in retrieval and Q-and-A performance over several existing baselines. Ablation studies further suggest that both memory components and their interaction mechanisms contribute to the overall performance.

Critical strengths

1.Memory is an important topic in LLM. The paper focuses on a well motivated problem. It is an important challenge of long-term memory retrieval for LLM agents, particularly when relevant information is distributed across multiple interactions and requires associative reasoning.

  1. The authors considered a well motivated method from human brain. The proposed Lumi-Mem framework combines sparse associative indexing with dense evidence retrieval through bidirectional interactions. The overall architecture is intuitive and provides a coherent approach to memory organization and retrieval.

  2. Strong empirical results: The method achieves competitive or superior retrieval and QA performance across three benchmarks. The ablation studies also provide useful evidence that the sparse and dense components, as well as their interaction mechanisms, contribute to the overall improvements.

Critical weaknesses

The main weakness I think is in the design and experiment.

  1. I think in its current form, the paper makes me not very easy to see what position the proposed method is in among some related/existing framework. Although the paper discusses a number of related methods, I find the positioning of Lumi-Mem relative to existing graph-based, associative, and hierarchical memory approaches insufficiently clear. In particular, it would be helpful to explain more explicitly what distinguishes the proposed sparse–dense interaction mechanism from existing approaches such as HippoRAG-v2 and other memory-enhanced retrieval frameworks.

  2. why Lumi-Mem, BKRAG, and GFM-RAG are trained on the same 80/20 query split, while the other baselines are evaluated in their original settings? Consequently, I am not fully convinced that the reported performance gains can be attributed solely to the proposed memory mechanism.

  3. Limited evaluation of long-term memory dynamics: Although the paper emphasizes memory self-organization, the memory bank is constructed offline. It doesn't consider continuous memory updates, evolving user information, or long-term interactions in a deployed agent, which is more practical.

  4. Insufficient analysis of efficiency and scalability: The proposed framework introduces multiple stages of memory construction, associative propagation, calibration, and retrieval. A more detailed comparison of end-to-end computational costs, memory maintenance overhead, and scalability would help establish whether the performance improvements justify the additional complexity.

Questions for authors

Please refer to the questions in Weakness 1 and 2.

4 comments

Thank you for the detailed feedback. Below we clarify how Lumi-Mem relates to prior work and how the baselines were evaluated, and report experiments on streaming updates, computational cost, and scalability. The corresponding changes are highlighted in blue in the revised manuscript.


W1. Positioning relative to related frameworks

Thank you for the comment. Below, we clarify our contributions and how we differ from existing memory-enhanced retrieval frameworks.

Our contribution and position. Inspired by the interaction between hippocampus and neocortex in human memory systems, Lumi-Mem develops bidirectional sparse–dense encoding to enable adaptive memory activation and self-organization for LLM agents. Specifically,

  • Adaptive Memory Activation. Our sparse level, the Sparse Attractor (SA), is inspired by how hippocampal activity settles into stable patterns. Related topics within SA excite each other, and inhibition limits the spread until activation settles. A learned adapter adjusts these dynamics to each query. In contrast, existing retrievals often apply fixed rules for memory activation. HippoRAG-v2 uses fixed Personalized PageRank, GFM-RAG uses a trained but fixed-depth graph network, and HeLa-Mem spreads activation over Hebbian links using a predefined rule.

  • Memory Self-Organization. Lumi-Mem introduces a bidirectional sparse–dense interaction. SA's topic activation guides chunk retrieval through a learned gate (sparse-to-dense). During training, chunk scores feed back to refine SA's topic selection (dense-to-sparse). These interactions align topic activation with chunk-level evidence for better memory self-organization. Other methods use one-way retrieval pipelines: dense chunk scores are summarized over graph to activate the memory (HippoRAG-v2), while graph-derived scores are used to retrieve final chunks (GFM-RAG).

Table A summarizes the key differences between Lumi-Mem and existing hierarchical, associative, and graph-based approaches.

Table A. Design features of Lumi-Mem and related memory approaches.

Approach (examples)Multi-level organizationGraph propagationAdaptive memory activationMemory self-organization
Memory systems (Mem0, A-Mem)––––
Hierarchical (RAPTOR, GraphRAG, MemoryOS)✓–––
Associative (HeLa-Mem)–✓––
Graph-based (HippoRAG, HippoRAG-v2, GFM-RAG)✓✓––
Lumi-Mem✓✓✓✓

Revised contents: Please see p. 3 (lines 136–139).


W2. Training split and baseline settings

Thanks for the comment.

Why the settings differ. All methods are evaluated on the same test queries. BKRAG, GFM-RAG, and Lumi-Mem are training-based; we therefore train all three on the same 80/20 train/test split for a matched comparison. BM25, HippoRAG-v2, and ComoRAG are training-free, so we directly evaluate them on the same test set.

Whether the gains come from the memory mechanism. Below, R@5 is the fraction of gold evidence found in the top five retrieved results.

  • Same training, different methods. Lumi-Mem outperforms the two baselines trained on the same split by 6.75–12.75 R@5 (Table 1).
  • No training at all. Without any training, Lumi-Mem reaches 84.75 R@5 on LongMemEval (Table 6). This is already above every baseline except PBR.
  • Same training, one component removed. Our ablations retrain the model on the same split and remove one component at a time (Table B). Removing each component lowers performance to a different degree. For example, removing the sparse topic memory alone lowers R@5 by 4.16–24.00 points with Contriever (Table 3).

The gains therefore come from the memory mechanism, not from training alone.

Table B. Description of ablation variants.

VariantWhat changesWhat it tests
w/o SARemove the sparse topic memory. Rank by dense retrieval alone.Whether sparse topic memory helps beyond dense retrieval.
w/o DRReplace the pretrained dense scorer with the retriever's own similarity.What dense-scorer pretraining adds.
w/o Adaptive GateCombine sparse and dense scores with one shared learned weight.Whether the weight should adapt to the query and chunk.
w/o Dense-to-Sparse CalibrationRemove the training signal that calibrates the sparse topic memory.Whether this calibration helps topic selection.
w/o Sparse-to-Dense ActivationReplace the learned sparse-to-dense score with a fixed graph-based score.Whether the learned mapping adds beyond fixed graph evidence.

Revised contents: Please see p. 7 (lines 356–360) and p. 23 (lines 1190–1209).

W3. Long-term memory dynamics

Thank you for raising this point. To test continuous updates you mentioned, we have added a streaming evaluation, including 78 update cases from LongMemEval and 75 from LoCoMo (a newly added long-term conversational memory benchmark). In each case, a piece of user information changes during the conversation. We replay the conversation in timestamp order, and each query retrieves only from what has already been written. We compare our incremental merging (Appendix D) with an append-only policy, which stores every memory item as a new topic. We report how often Lumi-Mem retrieves the updated information (R@5), and how often its answer still gives the outdated value.

With incremental merging, Lumi-Mem retrieves the updated information in 98.6% of LongMemEval cases and 81.3% of LoCoMo cases (Table C). Compared with appending, merging slightly improves retrieval and roughly halves the number of topics. Evaluating Lumi-Mem over longer interactions in a deployed agent is a direction for future work.

Table C. Retrieval under streaming memory updates.

DatasetWrite policyR@5 (%)Old-value answers (%)Mean topicsWrite time per update (s)
LongMemEvalAppend only97.212.3215.715.93
Incremental merge98.611.8116.4418.60
LoCoMoAppend only78.74.2120.884.57
Incremental merge81.34.250.6711.97

Revised contents: Please see p. 10 (lines 513–514) and pp. 26–27 (lines 1396–1411).


W4. Efficiency and scalability

Thank you for this suggestion. Lumi-Mem's cost is comparable to that of the baselines, while its accuracy is higher. We have compared the end-to-end cost of Lumi-Mem with HippoRAG-v2, BKRAG, and GFM-RAG on all three datasets using the same device (Table D).

  • Memory construction (one-time, offline). Lumi-Mem builds its memory bank in about half the time of HippoRAG-v2 and GFM-RAG. Only BKRAG builds faster.
  • Training (one-time). Lumi-Mem trains in 19–44 minutes, similar to BKRAG and GFM-RAG. HippoRAG-v2 needs no training.
  • Retrieval (per query). Lumi-Mem is faster than BKRAG and GFM-RAG, but slower than HippoRAG-v2. At this speed, it still reaches the highest R@5 of all methods (Table 1).
  • Memory maintenance overhead. Lumi-Mem stores a retrieval index of at most 535.2 MiB, similar in size to HippoRAG-v2's. Reducing this maintenance cost is a direction for future work.

Table D. Construction, training, retrieval, and storage cost of Lumi-Mem and baselines.

LongMemEval2WikiMultiHopQAMuSiQue
Construction time (h)
HippoRAG-v225.112.19.0
BKRAG8.10.81.4
GFM-RAG25.110.16.4
Lumi-Mem12.55.33.5
Training time (min)
HippoRAG-v2000
BKRAG47.116.921.3
GFM-RAG21.221.126.3
Lumi-Mem43.518.722.4
Retrieval latency (ms/query)
HippoRAG-v241.341.382.8
BKRAG130.2158.3206.3
GFM-RAG63.4105.6155.5
Lumi-Mem50.685.8148.4
Stored retrieval index (MiB)
HippoRAG-v2565.3268.2388.3
BKRAG542.4130.4154.7
GFM-RAG236.9381.4713.8
Lumi-Mem535.2232.7415.4

As for scalability, we have tested how accuracy and latency scale with bank size (Table E). As the memory bank grows three- to fourfold, Lumi-Mem's accuracy declines gradually and its latency grows about 2.5 times. HippoRAG-v2, tested on the same bank sizes, loses more R@5, and its latency grows about fourfold (Table F). Lumi-Mem keeps a higher R@5 at every bank size. Our future work will improve Lumi-Mem's robustness to distractors and its latency on larger memory banks.

Table E. Lumi-Mem retrieval accuracy and latency at four memory-bank sizes.

DatasetMemory nodesAccuracy (%)Latency (ms/query)
R@5/N@5Mean/p95
LongMemEval15,24195.92/84.2620.8/34.4
30,48393.42/79.5034.3/53.7
45,72490.92/76.1943.4/66.5
60,96690.58/74.0850.6/77.7
2WikiMultiHopQA15,90795.50/95.1434.3/51.8
34,19694.63/94.3856.3/80.7
49,19194.29/93.9372.3/101.8
63,55993.38/93.1885.8/121.1
MuSiQue32,99370.67/67.1657.2/82.5
56,94769.39/65.9285.0/119.6
82,37866.93/64.50116.9/160.8
106,00765.54/63.90148.4/205.1

Table F. HippoRAG-v2 retrieval accuracy and latency at four memory-bank sizes.

DatasetMemory nodesAccuracy (%)Latency (ms/query)
R@5/N@5Mean/p95
LongMemEval15,24189.50/82.6610.3/20.4
30,48384.42/75.2320.7/37.4
45,72480.50/70.3331.4/54.2
60,96680.58/67.9941.3/65.6
2WikiMultiHopQA15,90790.88/89.579.8/16.9
34,19687.33/86.6322.3/33.4
49,19186.04/85.6833.6/48.4
63,55985.25/85.2141.3/63.7
MuSiQue32,99367.88/67.9223.0/35.9
56,94765.71/65.6040.1/60.3
82,37863.67/64.0461.2/91.9
106,00762.63/62.3182.8/118.0

Revised contents: Please see pp. 21–22 (lines 1126–1186).

Thank you for the detailed response and additional experiments which mostly address my previous concerns. I thus will raise the score to 3 accordingly.

We are glad that our responses have addressed your concerns, and we greatly appreciate your decision to increase the score.

Reviewer_7b5mOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LUMI-MEM, a retrieval method for long-term LLM-agent memory inspired by the interaction between hippocampus and neocortex. An LLM first turns raw sessions or documents into a memory bank of topic "basins", each with a summary, an embedding and a set of evidence chunks. At query time a Sparse Attractor runs a few damped recurrent steps over a graph of these basins to get a topic posterior, a pretrained, frozen Dense Representation scores individual chunks, and a learned gate adds a chunk-level sparse-to-dense activation, computed from the topic posterior, to the dense score. During training a KL loss pulls the topic posterior toward a target that combines a gold-topic signal with dense feedback, which the authors call dense-to-sparse calibration. On LongMemEval, 2WikiMultiHopQA and MuSiQue the method reports the best R@5 on all three benchmarks and the best average QA F1 with GPT-4.1-mini, and the online sparse step takes a few milliseconds per query.

Critical strengths

The retrieval numbers are strong, and not only against systems evaluated in their original settings. The two baselines trained on the same split also trail by a wide margin, by more than twelve points of R@5 on 2Wiki and more than eight on MuSiQue (Table 1). LUMI-MEM still ranks first on almost every metric when Stella replaces Contriever (Table 8), and the online cost of the sparse step is small (Table 10). I also liked the matched comparison with personalized PageRank in Table 6, which keeps the candidates, graphs and fusion fixed and changes only the propagation rule. That is the right kind of control for a claim about propagation, even though it is run only on LongMemEval.

Critical weaknesses

My main reservation is about attribution: the results do not yet show that the hippocampus–neocortex machinery is what produces the gains. LUMI-MEM is trained with gold evidence on 80% of each benchmark's queries, in DR pretraining, in the ranking loss and in the calibration target, and it retrieves over an LLM-built topic bank. Stripped down, SA starts from dense retrieval over the LLM-written topic summaries and runs a few damped steps of a graph propagation whose inhibition is set above the largest eigenvalue of B(q)B(q) (Eq. 20), so every query has exactly one equilibrium (Appendix C.1). With a single equilibrium per query, SA behaves more like a query-conditioned graph propagation than an attractor memory with several stored states, and PPR, which the paper uses as its matched comparison, is the natural reference. Against PPR it gains about a point and a half of R@5 without labels and about five with supervision, with intervals that exclude zero (Table 6 and Appendix C.2), which is evidence for the propagation rule. On 2Wiki with Contriever, removing DR costs about one point of R@5 (Table 3), which suggests that most of the performance there comes through the topic bank, although, as noted below, the paper does not say how each ablation is implemented. Yet there is no control that scores the same topic summaries without propagation, that is, SA with T=0T=0. The dense part of the calibration target does not show a benefit either. In Table 7 the best setting on every dataset and metric uses the gold-topic signal alone, with no dense scores mixed in, although the differences from the default are a fraction of a point. Dense uncertainty still sets the calibration weight, so this does not rule out a role for the dense signal, but the evidence for it is thin. What the evidence supports is that an LLM-built topic bank with supervised topic scoring helps, and that SA's propagation rule does better than PPR. Whether the bidirectional sparse–dense interaction adds much beyond that is not clearly shown, and the half of the loop that feeds dense evidence back to the topics acts only during training, with the memory bank built offline. This does not make the results less useful, but the framing should follow the evidence, and the control in Question 2 would help separate the topic bank from the propagation.

The ablations are described only by what they remove, and I could not make two of them agree. Table 3 says only that "w/o SA removes SA", and Fig. 2 does not say what replaces the gate or the sparse-to-dense term. If removing sparse-to-dense activation means dropping Gate⋅ss2d\mathrm{Gate}\cdot s_{\text{s2d}} from Eq. 8, the score is the z-scored DR within CqC_q. Since CqC_q always contains the top DR chunks (Eq. 2), its top 5 are the global DR top 5, which is the same ranking as without SA if both variants use the same frozen DR checkpoint. Yet on 2Wiki with Contriever this variant is more than twenty points of R@5 above "w/o SA" (Fig. 2 against Table 3). If the ablation instead removes only the associative support ψassoc\psi_{\text{assoc}}, as the wording on pp. 7–8 might suggest, the paper should say so. The main results are also averages of five runs with no run-to-run spread for Tables 1–3, on test sets of one or two hundred queries, and several margins are under a point, for example MuSiQue F1 against ThinkQE (Table 2). Fig. 2 uses truncated y-axes, which makes small gaps look large.

Questions for authors
  1. Clarification: What exactly replaces each ablated component in Table 3 and Fig. 2? Do all variants share the same frozen DR checkpoint? In particular, does "w/o Sparse-to-dense Activation" drop the whole Gate⋅ss2d\mathrm{Gate}\cdot s_{\text{s2d}} term or only ψassoc\psi_{\text{assoc}}, and how does that square with the 69.38 R@5 for "w/o SA" on 2Wiki with Contriever?
  2. Clarification: How does scoring the same LLM-built topic summaries without propagation (T=0T=0) perform under the same supervision? This would separate the topic bank from the attractor dynamics.
  3. Clarification: The paper says all methods share the candidate pool. Does this also hold for the baselines evaluated in their original settings, and how are the gold chunks GqG_q mapped onto the LLM-extracted chunks?
  4. Clarification: Please report the spread over the five runs for Tables 1–3.
5 comments

Thank you for the careful review. Below we clarify the ablations and candidate pools, and report new evidence on the role of propagation, topic recall, and variability across runs. The corresponding changes are highlighted in blue in the revised manuscript.


W1. Attribution of the gains

Thank you for your careful analysis. To clarify where the gains come from, we first reinterpret the results already in the paper. Then, we add three new experiments: a comparison with DR pretraining alone, the T = 0 control you suggest, and the Topic Recall of SA. The evidence shows that the gains do not come from supervision or the topic bank alone. It also supports a narrower framing than the paper currently uses, and we have revised the paper accordingly.

Clarification of existing results.

  • Training with gold evidence. Supervision is not the main source of the gains. As shown in Table 6, even without any labels, Lumi-Mem outperforms every baseline on LongMemEval except PBR (84.75 R@5). With supervision, it outperforms the baselines trained on the same split by 6.75–12.75 R@5 (Table 1).

  • SA's contribution to the final retrieval. Since each query has a single equilibrium, SA is indeed a query-conditioned propagation. To isolate and evaluate its propagation rule, Table 6 replaces SA in Lumi-Mem with PPR (the propagation mechanism used in HippoRAG) while keeping all other components identical. SA consistently outperforms PPR in both settings, achieving gains of 1.42 R@5 without labels and 5.30 R@5 with supervision. We have stated this explicitly in Section 4.2.

  • Bidirectional interaction's contribution. Sparse-to-dense activation improves retrieval. Dense-to-sparse calibration acts during training, where it can replace gold-topic labels with only a small drop in accuracy. On 2Wiki, sparse-to-dense activation raises R@5 from 69.38 with DR alone to 93.38 (Table 3 and Fig. 2). Across all three datasets, training SA with dense feedback alone (α = 0) yields R@3 and N@3 scores 1.24–2.62 points below those obtained with gold-topic supervision alone (α = 1; Table 7).

Comparison with DR pretraining alone (new). To further address your concern, we have compared the full model with DR pretraining alone. The full model outperforms DR alone in all six dataset–retriever settings, by 1.95–24.15 R@5 (Table A).

Table A. Gains of the full model over DR pretraining alone, in R@5/N@5.

DatasetContrieverStella
LongMemEval+4.16 / +3.71+1.95 / +2.36
2Wiki+24.00 / +21.47+24.15 / +19.07
MuSiQue+13.58 / +9.32+20.08 / +21.97

Topic bank without propagation (new). As you suggested, we have tested the T = 0 control on LongMemEval. It scores the same LLM-built topic summaries without propagation, under the same supervision as Lumi-Mem. T = 0 reaches 83.33 R@5 and 71.70 N@5. This is 7.25 R@5 below the full model (95% CI [4.37, 9.10]). This result suggests that the recall gain comes from propagation, not from the topic bank alone.

Topic Recall of SA (new). We have measured Topic Recall on the test queries, i.e., the fraction of topics containing gold evidence that SA selects. With the same gold-topic labels, adding dense feedback (α = 0.8) raises Topic Recall significantly on all three datasets (Table B). While gold-topic labels treat all gold topics equally, DR feedback provides an additional relevance signal reflecting how strongly the retrieved chunks support each topic.

Table B. Topic Recall (%) on the test queries (higher is better).

Datasetα = 0.8α = 1Difference [95% CI]
LongMemEval97.0896.55+0.53 [0.13, 1.56]
2Wiki59.6258.64+0.98 [0.58, 2.03]
MuSiQue45.9345.02+0.91 [0.24, 2.03]

Revised framing. Together, the evidence supports three claims. The recall gain likely comes from propagation over the topic graph, not from the topic bank alone. The full model improves on the dense retriever beyond what training gives. Dense feedback acts only during training, where it mainly helps SA choose the right topics. We have revised the Introduction, Section 4.2, and Section 5.3 accordingly.

Revised contents: Please see p. 2 (lines 98–100), p. 4 (lines 178–179), p. 8 (lines 418–422), p. 19 (lines 1000–1001), and p. 27 (lines 1415–1444).

W2. Ablation definitions and run variability

Thank you for your careful reading. We have specified what replaces each ablated component. This explains why the two variants you compared give different results. For brevity, see Appendix C.9 of the revised paper (Tables 15–17, pp. 24–26) for the five-run means and standard deviations for Tables 1–3. We also tested the small margins with paired confidence intervals.

What each ablation replaces. Table C lists what each variant changes, what it keeps, and what it tests. We have provided this table in the revised version.

Table C. Description of ablation variants.

VariantWhat changesWhat it tests
w/o SARemove the sparse topic memory, i.e., the whole Gate⋅ss2d{Gate}·s^{{s2d}} term. Rank by DR alone.Whether sparse topic memory helps beyond DR.
w/o DRReplace the pretrained DR score with the retriever's cosine similarity, also in calibration.What DR score adds.
w/o Adaptive GateReplace the gate with one learned weight g₀ shared by all queries and chunks.Whether the weight should adapt to the query and chunk.
w/o Dense-to-Sparse CalibrationRemove the calibration loss Ld2sℒ_{{d2s}}. Keep the ranking and gate losses.Whether calibration helps topic selection.
w/o Sparse-to-Dense ActivationReplace the learned ss2ds^{{s2d}} with a fixed graph-derived score. Keep and refit the gate, whose features still include ψdirψ_{{dir}} and ψassocψ_{{assoc}}.Whether the learned score adds beyond fixed graph evidence.

Why w/o SA and w/o Sparse-to-Dense Activation differ. The two variants remove different parts, so the gap between them is expected. w/o SA removes the whole gated term Gate⋅ss2d{Gate}·s^{{s2d}}. Only DR remains. w/o Sparse-to-Dense Activation only replaces the learned ss2ds^{{s2d}} with a fixed graph-derived score. It keeps the gate and its graph features, including ψassocψ_{{assoc}}, so graph evidence still enters the score. This is why it scores much higher than w/o SA. We have stated each replacement in the paper.

Run-to-run spread (new). Lumi-Mem's R@5 leads over the strongest baseline (2.91–8.13 points) far exceed its standard deviation (at most 0.87). BM25, Contriever, and Stella are deterministic, so their standard deviation is zero.

Small margins (new). To check the small margins on these test sets, we have also computed paired 95% CIs over the test questions for all retrieval and QA metrics (Table D). Lumi-Mem's improvement over the strongest baseline is significant on 14 of 18 metrics and never significantly negative. Particularly, the MuSiQue F1 lead over ThinkQE (+0.64) is significant (CI [0.17, 1.51]). The LongMemEval F1 lead (+0.25) is not significant (CI [-0.34, 1.07]), and we have described it as comparable.

Table D. Paired 95% CIs of Lumi-Mem's improvement over the strongest baseline.

MetricLongMemEval2WikiMuSiQue
Retrieval
R@3+2.91 [0.85, 4.97]+6.25 [4.18, 8.32]-0.63 [-1.46, 0.20]
R@5+4.00 [1.57, 6.43]+8.13 [5.92, 10.34]+2.91 [1.87, 3.95]
N@3+5.46 [4.02, 6.90]+7.18 [5.02, 9.34]+0.62 [-0.77, 2.01]
N@5+5.95 [4.49, 7.41]+7.97 [6.07, 9.87]+1.59 [0.29, 2.89]
QA
EM-1.00 [-3.63, 1.63]+5.00 [2.08, 7.92]+1.50 [0.31, 2.69]
F1+0.25 [-0.34, 1.07]+6.02 [3.31, 8.73]+0.64 [0.17, 1.51]

Fig. 2 values. The truncated y-axes in Fig. 2 can make the results unclear, so Table E lists the value behind each bar.

Table E. Values in the paper's Fig. 2 (R@5 / N@5).

VariantRetrieverLongMemEval2Wiki
FullContriever90.58/74.0893.38/93.18
Stella84.70/69.2089.90/90.20
w/o Adaptive GateContriever87.83/71.9973.13/76.71
Stella84.08/68.4869.25/74.54
w/o Dense-to-Sparse CalibrationContriever90.18/73.0892.00/92.45
Stella84.37/68.2287.25/88.53
w/o Sparse-to-Dense ActivationContriever88.33/73.3291.20/92.43
Stella84.00/68.5786.63/87.56

Revised contents: Please see p. 7 (lines 325–326), p. 8 (lines 380–381, 408–409, and 430–431), p. 9 (lines 435–436 and 458–461), and pp. 23–26 (lines 1188–1367).

Q1. Ablation components and shared DR checkpoint

All variants share the same frozen DR checkpoint, except w/o DR, which replaces its score with the retriever's cosine similarity. w/o Sparse-to-Dense Activation keeps ψassocψ_{{assoc}} and the gated term Gate⋅ss2d{Gate}·s^{{s2d}}. It only replaces the learned ss2ds^{{s2d}} inside the term with a fixed graph-derived score, and ψassocψ_{{assoc}} stays as a gate feature. Graph evidence therefore still enters its score. This variant tests whether the learned score adds beyond this fixed graph evidence. w/o SA removes the sparse topic memory, i.e., the whole gated term, including the gate and its graph features. As a result, R@5 for w/o SA drops to 69.38 on 2Wiki with Contriever. Please kindly refer to Table C in the response to Weakness 2 for what replaces each component in detail.

Revised contents: Please see p. 8 (lines 430–431), p. 9 (lines 432–434 and 442), and p. 23 (lines 1190–1209 and 1217–1222).


Q2. Topic summaries without propagation

Without propagation, Lumi-Mem's retrieval degrades substantially. We have run this control on LongMemEval. The variant without propagation keeps the same topic bank, candidate set, and supervision as Lumi-Mem, and removes only propagation. T = 0 reaches 83.33 R@5, which is 7.25 points below full model (95% CI [4.37, 9.10]). This result suggests that the gain comes from propagation, not from the topic bank alone. Please kindly refer to the response to Weakness 1 for how this control fits with our other new results.

Revised contents: Please see p. 19 (lines 1000–1001) and p. 27 (lines 1416–1420).


Q3. Candidate pools and gold-chunk mapping

Thanks for the comment.

Candidate pool. All methods we used share the same test queries, corpus, and metrics. Our ablations and matched comparisons in Tables 3 and 6 share one candidate pool (the full model's 𝒞q𝒞_{{q}} in Eq. 2). Baselines in Table 1 keep their own memory construction and retrieval pipelines, so they build their own candidates. We have corrected the sentence in the paper that suggests otherwise.

Gold chunk mapping. As for mapping process, we adopt a source-ID-based mapping. The benchmarks label gold evidence on the original source units, such as dialogue turns or passages. We map these labels to our LLM-extracted chunks in two steps.

  1. When the memory bank is built, each LLM-extracted chunk records the IDs of the source units it comes from.
  2. A chunk belongs to the gold chunks 𝒢q𝒢_{{q}} if any of its source units is labeled as gold for query q.

Revised contents: Please see p. 3 (lines 157–159) and p. 7 (lines 356–360).


Q4. Spread over five runs

The five-run means and standard deviations for Tables 1–3 are in Appendix C.9 of the revised paper (Tables 15–17, pp. 24–26). Lumi-Mem's R@5 leads over the strongest baseline (2.91–8.13 points) far exceed its standard deviation (at most 0.87), so its retrieval gains are not due to run-to-run variation.

Revised contents: Please see p. 7 (lines 325–326), p. 8 (lines 380–381), p. 9 (lines 435–436), and pp. 24–26 (lines 1244–1367).

Reviewer_PHiZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LUMI-MEM, which combines topic-level associative retrieval with chunk-level dense scoring. A trainable attractor propagates query activation over a topic graph, while an adaptive gate combines topic support with dense relevance. Experiments on three benchmarks show strong retrieval results and improved average QA performance, supported by ablations and transfer experiments.

Critical strengths
  1. The method offers a concrete way to combine topic-level associations with fine-grained evidence matching. Learning the propagation controls and fusion gate makes the interaction more flexible than simply combining two retrieval scores.
  2. The shared adapter avoids learning separate parameters for each topic basin. Once DR is pretrained and frozen, joint training updates only the adapter, sparse-to-dense weights, and gate, allowing the learned components to be applied to new memory banks without basin-specific fitting.
Critical weaknesses
  1. The comparisons do not clearly separate the benefit of the proposed architecture from task-specific training, particularly when several baselines are evaluated in their original settings. Although Table 6 includes label-free and supervised variants, their training differences are not sufficiently clear; a matched trained-versus-untrained comparison would help establish how much training contributes.

  2. The benchmark coverage is somewhat narrow for a paper focused on long-term agent memory, with two of the three datasets evaluating document-based multi-hop QA. Including LoCoMo and BEAM would help assess whether the gains extend to more diverse conversational memory tasks and longer interaction histories.

Questions for authors
  1. Could you clarify which components are trained in the label-free and supervised settings in Table 6? A comparison between DR pretraining alone and the full training pipeline would help identify how much the learned sparse components add.

  2. How should we interpret the α = 1 results relative to the motivation for bidirectional sparse–dense interaction? Are there settings, such as limited supervision or cross-corpus transfer, where retaining dense feedback is particularly helpful?

2 comments

Thank you for your feedback. Below we try our best to address your concerns. The corresponding changes are highlighted in blue in the revised manuscript.


W1

Thank you for your comment. To separate the benefit of the architecture from training, we first clarify the training setting in Tables 1 and 6.

Training of baselines. Existing memory systems can be constructed using either training-based or training-free approaches. Lumi-Mem is a training-based method. Nevertheless, to comprehensively evaluate its performance, we compare Lumi-Mem against both training-free and training-based baselines. As shown in Table 1, the specific settings and related results are:

  • Training-free: BM25, PBR, ThinkQE, HippoRAG, and ComoRAG have no parameters to fit. We therefore run them in their original settings. The results show that Lumi-Mem improves R@5 by 2.91–8.13 points over the best training-free baseline on each dataset.

  • Training-based: BKRAG, GFM-RAG, and Lumi-Mem have trainable components. All three are trained on the same 80/20 query split. The results show that Lumi-Mem improves R@5 by 6.75–12.75 points over the stronger trained baseline on each dataset.

Trained versus untrained Lumi-Mem. Table 6 provides the results of untrained (label-free) Lumi-Mem and compares them with those of our full model. For comparison, both settings share the memory bank, topic graphs, and propagation settings and are evaluated on the same queries from the unseen dataset. They differ only in whether parameters are fit to gold evidence. Specifically,

  • Untrained: no parameters are fit. The dense retriever (DR) uses the retriever's standardized similarity. The sparse-to-dense weights and the gate are set to fixed values, and SA's adapter is not trained.

  • Trained: DR is pretrained on the training split and then frozen. The sparse-to-dense weights, the gate, and SA's adapter are then trained.

The results show that training raises R@5 from 84.75 to 90.58 and N@5 from 68.06 to 74.08. Even without training, Lumi-Mem's R@5 on LongMemEval is above all baselines in Table 1 except PBR (86.58). This suggests that the architecture itself, not only training, contributes to the gains.

Comparison with DR pretraining alone (new). To further verify the gains of our architecture, we have added a comparison between our full model and DR pretraining alone. The full model outperforms DR alone in all six dataset–retriever settings, by 1.95–24.15 R@5 (Table A). Our gain of architecture therefore does not come from DR pretraining alone. We have added this table to the appendix.

Table A. Gains of the full model over DR pretraining alone, in R@5/N@5.

DatasetContrieverStella
LongMemEval+4.16 / +3.71+1.95 / +2.36
2Wiki+24.00 / +21.47+24.15 / +19.07
MuSiQue+13.58 / +9.32+20.08 / +21.97

Revised contents: Please see p. 7 (lines 356–360), p. 8 (lines 421–422), p. 19 (lines 977–982 and 1000–1001), and p. 27 (lines 1422–1431).


W2

Thank you for your suggestion. To address your concern, we have introduced LoCoMo and BEAM-128K as two additional long-term memory benchmarks. LoCoMo has 1,527 questions over 10 conversations of 16.6K tokens each, while BEAM-128K has 355 questions across 20 much longer conversations of 129.8K tokens each. As shown in Table B, Lumi-Mem consistently outperforms the strongest baseline across all evaluated metrics, with improvements ranging from 6.03 to 8.36 points on every metric under five-fold cross-validation.

Table B. Retrieval results (%) on LoCoMo and BEAM-128K. Metric order in each cell is R@3 / R@5 / N@3 / N@5.

DatasetMethodMeanStd
LoCoMoBKRAG4̲2̲.̲5̲9̲ / 4̲7̲.̲9̲5̲ / 3̲8̲.̲3̲7̲ / 4̲0̲.̲6̲9̲7.44 / 6.48 / 6.37 / 5.99
GFM-RAG34.40 / 43.36 / 29.64 / 33.418.88 / 8.56 / 7.16 / 7.02
ComoRAG33.68 / 42.49 / 28.02 / 31.878.52 / 8.87 / 7.27 / 7.39
HippoRAG-v237.87 / 45.36 / 33.61 / 36.706.75 / 7.60 / 5.67 / 5.89
Ours50.06 / 56.31 / 44.40 / 47.186.79 / 5.11 / 5.14 / 4.44
BEAM-128KBKRAG29.88 / 36.78 / 25.26 / 28.452.90 / 3.44 / 2.41 / 3.02
GFM-RAG31.99 / 41.23 / 28.13 / 31.645.48 / 5.16 / 4.40 / 4.14
ComoRAG3̲5̲.̲4̲1̲ / 4̲2̲.̲2̲7̲ / 3̲0̲.̲5̲8̲ / 3̲3̲.̲4̲0̲1.84 / 3.35 / 1.38 / 1.48
HippoRAG-v228.66 / 38.16 / 23.86 / 27.862.15 / 3.96 / 1.66 / 1.96
Ours41.95 / 49.39 / 37.61 / 40.091.34 / 2.73 / 1.60 / 1.81

Revised contents: Please see p. 6 (lines 319–320), p. 10 (line 513), and p. 26 (lines 1371–1392).

Q1. Training in Table 6

The specific settings in Table 6 are as below.

  • Label-free: no parameters are fit.
  • Supervised: DR is pretrained on the training split and then frozen. The sparse-to-dense weights, the gate, and SA's adapter are then trained.

To further address your concern, we have compared the full model with DR pretraining alone. The full model outperforms DR alone in all six dataset–retriever settings, by 1.95–24.15 R@5. Please kindly refer to the response to Weakness 1 for details (Table A).

Revised contents: Please see p. 19 (lines 977–982 and 1000–1001) and p. 27 (lines 1422–1431).


Q2. Interpreting α = 1

Although the setting without dense feedback (α = 1) gives the best top-three ranking, dense feedback still helps SA select the topics containing gold evidence.

Dense feedback helps SA choose topics. We have measured Topic Recall on the test queries, i.e., the fraction of topics containing gold evidence that SA selects. With the same gold-topic labels, adding dense feedback (α = 0.8) raises Topic Recall significantly on all three datasets (Table C). While gold-topic labels treat all gold topics equally, DR feedback provides an additional relevance signal reflecting how strongly the retrieved chunks support each topic.

Table C. Topic Recall (%) on the test queries (higher is better).

Datasetα = 0.8α = 1Difference [95% CI]
LongMemEval97.0896.55+0.53 [0.13, 1.56]
2Wiki59.6258.64+0.98 [0.58, 2.03]
MuSiQue45.9345.02+0.91 [0.24, 2.03]

Dense feedback helps under limited supervision. Limited supervision is one such setting, as you suggest. Without any gold-topic labels (α = 0), R@3 and N@3 are only 1.24–2.62 points below α = 1 (Table 7). Gold-topic labels require annotated evidence. When such annotations are missing, SA can still be trained with dense feedback alone to substitute for gold labels, at a small cost. This substitution has also been explored in existing retriever training work (Izacard & Grave, ICLR 2021; Wang et al., NAACL 2022; Shi et al., NAACL 2024).

Revised contents: Please see p. 19 (lines 1020–1025) and p. 27 (lines 1434–1444).

Reviewer_Z7MBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper proposes a new memory retrieval framework called LUMI-MEM. It takes inspiration from the interactions between the hippocampus and neocortex. LUMI-MEM helps LLM agents retrieve evidence that is scattered across either conversations or documents. First, using GPT-4o mini, it extracts memories along with their supporting evidence (stored as evidence chunks). Then, with a second prompt, it groups the related memories into relevant topics, which the paper calls "topic basins". Given a question, the retrieval procedure follows two complementary routes. Sparse Attractor (SA) finds and ranks relevant topics using semantic and metadata links, while Dense Representation (DR) scores the individual chunks directly. The chunks from the top-ranked topics and the chunks ranked highest by DR are combined to form the candidate set. Then, a learned gate controls how much the topic support should affect each chunk’s dense score before producing the final ranking. The method is evaluated on LongMemEval, 2WikiMultiHopQA and MuSiQue, using 80/20 query splits (100, 200 and 200 test questions, respectively). It is compared against various baselines, including direct and enhanced retrievers and memory-based systems. Compared with these baselines, the paper reports the best R@5 on all three datasets and the best average answer F1 performance on QA with GPT-4.1-mini. It also reports ablations with two retrievers that the authors present as evidence that each component contributes.

Critical strengths
  1. The paper addresses an important and timely retrieval problem for LLM agents because answering a single question can require evidence from several conversations or documents, and finding only part of that evidence may be insufficient. The authors examine this problem in two settings, using LongMemEval to test how well the method recovers evidence from a user’s conversation history and 2WikiMultiHopQA and MuSiQue to test whether it can connect facts across documents. The coverage of both tasks and the comparison with a broad range of baselines, including recent memory systems, strengthen the evaluation (Section 5.1).
  2. The full system achieves the highest reported Recall@5 on all three datasets. The gains over the best baseline are particularly substantial on 2Wiki, with 93.38 vs 85.25 for Recall@5 and 69.62 vs 63.60 answer F1 (Tables 1-2). So overall, we can see that the retrieval advantage carries through to the downstream QA task which is also important.
Critical weaknesses
  1. Table 3 shows that removing SA lowers Recall@5 on all three datasets (with larger drops on the two multi-hop QA datasets) but it is less clear how much the proposed two way interaction between SA and DR contributes to the performance. In Figure 2, the calibration ablation removes the topic-level calibration loss, whose target combines DR feedback with gold-topic supervision. The resulting drop could therefore come from losing the gold label component of that loss alone. Table 7 compares the two signals a bit more directly. On all three datasets, gold-topic supervision alone gives slightly higher Recall@3 and NDCG@3 than the default mixture that includes DR feedback (gold-topic weight α=0.8 in Table 4). This leaves unclear whether adding DR feedback to gold-topic supervision helps SA learn which topics to prioritize. This matters because feedback from DR to SA is one direction of the proposed two way interaction (Section 5.3 and Appendix C.3).
  2. The paper claims the highest Recall@5 on all three datasets and the best average QA performance among the methods compared (Abstract, Tables 1 and 2). These results are based on the 100 test questions on LongMemEval and 200 each on 2Wiki and MuSiQue (Section 5.1 and Table 10). Earlier multi-hop QA studies (e.g., HippoRAG 2 and GFM-RAG) used larger test sets of 1000 questions per benchmark on 2Wiki and MuSiQue. This matters when interpreting the reported gains, especially the small answer F1 leads over the strongest baselines on LongMemEval (0.25 points) and MuSiQue (0.64 points). Section 5.1 states that results are averaged over five runs, but Table 2 gives no paired confidence intervals for these F1 differences, making it difficult to judge how sensitive the gains are to the choice of test questions.
Questions for authors

Main questions:

  1. The default setting (α=0.8) and the gold-only setting (α=1) in Table 7 use the same gold-topic labels during training. The default also includes DR feedback in the calibration target. Given that the gold-only setting gives slightly higher Recall@3 and NDCG@3, did adding DR feedback help SA select a larger fraction of the topics containing gold evidence on the test questions?
  2. Section 5.1 states that results are averaged over five runs. Which steps were repeated for LUMI-MEM and for each baseline (training, retrieval, or answer generation) when reporting the main results? Using the per-question results, could you report paired 95% confidence intervals for the main comparisons in Tables 1 and 2, especially the F1 leads on LongMemEval (+0.25) and MuSiQue (+0.64), as you did for the retrieval comparison in Appendix C.2?
  3. In the “w/o DR” variant of the ablation study in Table 3, did SA still receive DR feedback during training, with DR removed only at inference, or was that feedback also removed during training? Also, how were candidate chunks selected and ranked in this variant? And in Figure 2 what replaced the learned gate when adaptive gating was disabled?
  4. In Appendix A.4, the paper defines three types of associative edges based on shared sessions, creation dates and updates. Which of these actually produced links on 2Wiki and MuSiQue?

Minor comments (do not affect my assessment):

  • Figure 4 is shown in the main paper but is never explicitly mentioned in the text. It’s probably meant to be referenced in Section 5.5, where the authors discuss the results by question type.
  • Section 5.1 lists 6,119 associated passages for 2Wiki and 11,656 for MuSiQue. Table 10 reports the same count for MuSiQue but lists 6,120 chunks for 2Wiki. Does the preprocessing described in Appendix D account for the extra chunk on 2Wiki or is it a typo?

Suggestions for future work (do not affect my assessment):

  • Report memory-construction cost (LLM tokens and time per dataset), bank statistics (number of basins and edges), training time, and baseline latencies in Table 10.
  • Evaluate how accuracy and latency scale with bank size.
  • Consider also citing and discussing Panini: Continual Learning in Token Space via Structured Memory (ICML 2026; arXiv:2602.15156, https://arxiv.org/abs/2602.15156), which also draws on complementary learning systems theory for non-parametric LLM memory and evaluates on 2WikiMultiHopQA and MuSiQue against HippoRAG 2.
5 comments

Thank you for the detailed review and suggestions. Below we clarify the role of dense feedback and the ablations, and report new results on topic recall, paired confidence intervals, and additional benchmarks. The corresponding changes are highlighted in blue in the revised manuscript.


W1

Thank you for your comment. As you noted, gold-topic supervision alone in Table 7 yields slightly higher chunk retrieval performance. However, DR feedback can replace the effect of gold-topic labels at a small cost. Without any gold-topic labels (α = 0), R@3 and N@3 are only 1.24–2.62 points below the gold-only setting. This result indicates that Lumi-Mem's own relevance signal can effectively substitute for gold labels. This substitution has also been explored in existing retriever training work (Izacard & Grave, ICLR 2021; Wang et al., NAACL 2022; Shi et al., NAACL 2024).

To further address your concern, we have measured Topic Recall on the test queries, i.e., the fraction of topics containing gold evidence that SA selects. As shown in Table A, incorporating DR feedback into gold-topic supervision significantly improves Topic Recall across all three datasets. While gold-topic labels treat all gold topics equally, DR feedback provides an additional relevance signal reflecting how strongly the retrieved chunks support each topic.

Table A. Topic Recall (%) on the test queries (higher is better).

Datasetα = 0.8α = 1Difference [95% CI]
LongMemEval97.0896.55+0.53 [0.13, 1.56]
2Wiki59.6258.64+0.98 [0.58, 2.03]
MuSiQue45.9345.02+0.91 [0.24, 2.03]

Revised Contents: Please see p. 8 (lines 427–430), p. 19 (lines 1020–1023), and p. 27 (lines 1434–1444).


W2

Thank you for your comment. We have reported the paired confidence interval and expanded our evaluation to larger test sets. The results demonstrate that Lumi-Mem is not sensitive to the choice of test questions.

First, we have supplemented the results of paired CIs, as shown in Table B below. The results of paired 95% CIs over test questions show that Lumi-Mem significantly outperforms the strongest baseline on 14 of 18 metrics, with no statistically significant degradation on any metric. For retrieval performance, statistically significant gains are observed on 10 of 12 metrics. For QA, the gains on 2Wiki and MuSiQue are significant, while performance on LongMemEval is comparable to that of the strongest baseline.

Table B. Paired 95% CIs of Lumi-Mem's improvement over the strongest baseline.

MetricLongMemEval2WikiMuSiQue
Retrieval
R@3+2.91 [0.85, 4.97]+6.25 [4.18, 8.32]-0.63 [-1.46, 0.20]
R@5+4.00 [1.57, 6.43]+8.13 [5.92, 10.34]+2.91 [1.87, 3.95]
N@3+5.46 [4.02, 6.90]+7.18 [5.02, 9.34]+0.62 [-0.77, 2.01]
N@5+5.95 [4.49, 7.41]+7.97 [6.07, 9.87]+1.59 [0.29, 2.89]
QA
EM-1.00 [-3.63, 1.63]+5.00 [2.08, 7.92]+1.50 [0.31, 2.69]
F1+0.25 [-0.34, 1.07]+6.02 [3.31, 8.73]+0.64 [0.17, 1.51]

To further address your concern, we introduce LoCoMo and BEAM-128K as two additional long-term memory benchmarks. LoCoMo has 1,527 questions over 10 conversations of 16.6K tokens each, while BEAM-128K has 355 questions across 20 much longer conversations of 129.8K tokens each. As shown in Table C, Lumi-Mem consistently outperforms the strongest baseline across all evaluated metrics, with improvements ranging from 6.03 to 8.36 points on every metric under five-fold cross-validation.

Table C. Retrieval results (%) on LoCoMo and BEAM-128K. Metric order in each cell is R@3 / R@5 / N@3 / N@5 (higher is better).

DatasetMethodMeanStd
LoCoMoBKRAG4̲2̲.̲5̲9̲ / 4̲7̲.̲9̲5̲ / 3̲8̲.̲3̲7̲ / 4̲0̲.̲6̲9̲7.44 / 6.48 / 6.37 / 5.99
GFM-RAG34.40 / 43.36 / 29.64 / 33.418.88 / 8.56 / 7.16 / 7.02
ComoRAG33.68 / 42.49 / 28.02 / 31.878.52 / 8.87 / 7.27 / 7.39
HippoRAG-v237.87 / 45.36 / 33.61 / 36.706.75 / 7.60 / 5.67 / 5.89
Lumi-Mem (ours)50.06 / 56.31 / 44.40 / 47.186.79 / 5.11 / 5.14 / 4.44
BEAM-128KBKRAG29.88 / 36.78 / 25.26 / 28.452.90 / 3.44 / 2.41 / 3.02
GFM-RAG31.99 / 41.23 / 28.13 / 31.645.48 / 5.16 / 4.40 / 4.14
ComoRAG3̲5̲.̲4̲1̲ / 4̲2̲.̲2̲7̲ / 3̲0̲.̲5̲8̲ / 3̲3̲.̲4̲0̲1.84 / 3.35 / 1.38 / 1.48
HippoRAG-v228.66 / 38.16 / 23.86 / 27.862.15 / 3.96 / 1.66 / 1.96
Ours41.95 / 49.39 / 37.61 / 40.091.34 / 2.73 / 1.60 / 1.81

Revised Contents: Please see p. 6 (lines 319–320), p. 7 (lines 373–374), p. 8 (lines 408–409), p. 10 (lines 512–513), p. 24 (lines 1248–1264), and p. 26 (lines 1371–1392).

Q1. Gold-topic recall under DR feedback

Adding DR feedback to gold-topic supervision can improve SA's ability to select topics containing gold evidence across all three datasets. While gold-topic supervision treats all gold topics equally, DR feedback provides an additional signal reflecting how strongly the retrieved chunks support each topic. Please kindly refer to the response to Weakness 1 (Table A). Incorporating DR feedback into gold-topic supervision significantly improves Topic Recall across all three datasets.

Revised contents: Please see p. 27 (lines 1434–1444).


Q2. Repeated steps and confidence intervals

Each of the five runs uses a different random seed, with answers regenerated by the LLM based on the retrieval results of each run. For methods requiring training (Lumi-Mem, BKRAG, and GFM-RAG), the random seed affects training, retrieval, and answer generation. For training-free methods, the seed affects retrieval and answer generation. For methods with deterministic retrieval, such as BM25 and Contriever, the seed influences only LLM answer generation.

We have also supplemented paired 95% CIs to Tables 1 and 2. Lumi-Mem significantly outperforms the strongest baseline on 14 of 18 metrics. Specifically, the F1 gain on MuSiQue (+0.64) is significant (95% CI [0.17, 1.51]), whereas that on LongMemEval (+0.25) is not (CI [-0.34, 1.07]). Please kindly refer to our response to Weakness 2 (Table B) for comparison results.

Revised contents: Please see p. 7 (lines 325–326 and 363–366), p. 8 (lines 380–381), and p. 24 (lines 1253–1264).


Q3. w/o DR and adaptive-gate ablations

In w/o DR, the pretrained DR is replaced during both training and inference. During training, SA receives the feedback computed from the retriever's cosine similarity (standardized by Eq. 3), rather than the pretrained DR.

Selection. The w/o DR variant reuses the candidate chunks selected by the full model. These include the pretrained DR's top chunks and chunks from the topics selected by SA. Fixing the candidates lets both models rank the same chunks for a matched comparison.

Ranking. Within these candidates, chunks are ranked as in the full model, except that the cosine similarity replaces the DR score. This setting allows us to examine the impact of DR score on retrieval.

Adaptive gating. When adaptive gating is disabled in Fig. 2, the learned gate is replaced by a single learned weight shared by all queries and chunks. Consequently, graph evidence receives the same weight everywhere, allowing us to assess whether the weight needs to adapt to each query and chunk.

Revised contents: Please see p. 7 (lines 359–360), p. 9 (lines 432–434 and 458–461), and p. 23 (lines 1190–1205).


Q4. Associative edges on QA datasets

On 2Wiki and MuSiQue, only shared-session edges produce links. Since these benchmarks do not contain dialogue sessions, we treat each source passage as a session, linking two topics when their evidence comes from the same passage. Creation-date and update edges are not available for 2Wiki and MuSiQue, because their passages carry no timestamps.

Revised contents: Please see p. 15 (lines 768–771).


Q5. Figure 4 reference

Thank you for pointing this out. Fig. 4 presents the question-type comparison discussed in Section 5.5. We have added the missing reference to the opening sentence of that section.

Revised contents: Please see p. 9 (lines 479–480).


Q6. 2Wiki passage count

Thank you for pointing this out. This is a typo in Section 5.1. We consistently use 6,120 passages for 2Wiki, as reported in Table 10, and have corrected the number in Section 5.1 from 6,119 to 6,120.

Revised contents: Please see p. 6 (lines 317–318).

Q7. Cost and bank statistics

Thank you for this suggestion. We have added these item into Table D to place Lumi-Mem's latency next to the baselines for direct comparison. Lumi-Mem enjoys the advantage of low computational overhead, with SA propagation and signal fusion taking only 2.25–5.11 ms per query. However, its retrieval latency is higher than baselines, primarily due to candidate construction. Its memory bank also needs a one-time offline construction. Reducing the cost of candidate construction and memory-bank building is a direction for future work.

Table D. Memory-construction cost, bank statistics, and retrieval latency.

LongMemEval2WikiMultiHopQAMuSiQue
LLM tokens (M)141.020.914.4
Construction time (h)12.55.33.5
Basins (topic or phrase nodes)60,96663,559106,007
Basin edge records7,591,3901,164,0002,786,232
Training time (min)441922
Retrieval latency (ms/query)
BM250.4917.9964.32
HippoRAG-v131.2955.5797.71
HippoRAG-v241.2841.3382.84
Ours50.5685.81148.36

Revised contents: Please see pp. 21–22 (lines 1126–1161).


Q8. Bank-size scaling

Thank you for your suggestion. We have tested how accuracy and latency scale with bank size (Table E). As the memory bank grows three- to fourfold, accuracy declines gradually and latency grows about 2.5 times. Further improving Lumi-Mem's robustness to distractors and its latency on larger memory banks remains an important direction for future work.

Table E. Retrieval accuracy and latency at four memory-bank sizes.

DatasetTotal basinsR@5 (%)N@5 (%)Mean latency (ms/query)p95 latency (ms/query)
LongMemEval15,24195.9284.2620.8034.36
30,48393.4279.5034.3353.74
45,72490.9276.1943.4066.51
60,96690.5874.0850.5677.72
2WikiMultiHopQA15,90795.5095.1434.3451.76
34,19694.6394.3856.2580.69
49,19194.2993.9372.31101.79
63,55993.3893.1885.81121.12
MuSiQue32,99370.6767.1657.2482.47
56,94769.3965.9284.99119.61
82,37866.9364.50116.94160.83
106,00765.5463.90148.36205.06

Revised contents: Please see p. 22 (lines 1164–1186).


Q9. Panini citation

Thank you for pointing this work out, and we have cited and discuss Panini in the revised manuscript. Actually, Panini and Lumi-Mem differ in emphasis and are complementary: Panini structures experiences at write time into a network of question–answer pairs and retrieves inference chains instead of chunks, whereas ours keeps evidence chunks and focuses on how sparse topic memory and dense retrieval interact at read time.

Revised contents: Please see p. 3 (lines 131–133).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal and changes for AC

Dear Area Chair,

We thank the reviewers for their constructive feedback. We have highlighted the changes in blue in the revised manuscript. We summarize our rebuttal and changes below.

1. Sparse propagation and ablations (Z7MB, 7b5m, ka21). We have clarified query-conditioned topic activation in SA and specified what each ablation changes. With the topic bank, candidates, and supervision fixed, removing propagation on LongMemEval yields 83.33 R@5, 7.25 points below the full model (95% CI [4.37, 9.10]).

2. Training and evaluation (PHiZ, 7b5m, ka21). We have clarified the shared 80/20 split for BKRAG, GFM-RAG, and Lumi-Mem and when candidate pools are held fixed. The full model gains 1.95–24.15 R@5 points over dense retriever pretraining alone across six dataset–retriever settings.

3. Dense feedback (Z7MB, PHiZ, 7b5m). We have measured Topic Recall with the same gold-topic labels. Adding dense feedback improves Topic Recall by 0.53, 0.98, and 0.91 percentage points on LongMemEval, 2Wiki, and MuSiQue, respectively, with all paired 95% CIs above zero.

4. Benchmark coverage (Z7MB, PHiZ). We have added LoCoMo and BEAM-128K as new long-term memory benchmarks. In five-fold cross-validation, Lumi-Mem gains 6.03–8.36 points over the strongest baseline across retrieval metrics.

5. Statistical uncertainty (Z7MB, 7b5m). We have reported five-run standard deviations for Tables 1–3 and paired 95% CIs for Tables 1–2. Gains are significant on 14 of 18 metrics. The MuSiQue F1 gain is +0.64 [0.17, 1.51], and the LongMemEval F1 gain is +0.25 [−0.34, 1.07].

6. Streaming updates and efficiency (Z7MB, ka21). We have tested streaming updates, reported costs, and measured accuracy and latency at four bank sizes. Our incremental merging retrieves updated information in 98.6% of LongMemEval cases and 81.3% of LoCoMo cases, using roughly half as many topics as append-only storage.

As the discussion period has only one day remaining, we would really appreciate it if you could help encourage further discussion around our responses.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.