acceptodds
Under review as a conference paper at ICLR 2027

Memini: Continual Learning through Associative Memory in LLM Systems

30 views

Abstract

An LLM system can preserve every incoming document yet lose the ability to use earlier knowledge because new and old evidence compete within finite retrieval and context budgets. We introduce Memini, an external associative memory for continual learning from streams of unlabeled textual observations with all pretrained components held fixed. Drawing on episodic binding and cue-driven recall, Memini preserves each observation as an immutable source episode, while query-independent updates bind recurring concepts to episodes and accumulate source-attributable co-observations across them. At query time, partial cues follow bounded associative routes to recover complete source episodes, either as a standalone readout or in combination with dense retrieval. Reading does not modify persistent memory. On streams constructed from MuSiQue, 2WikiMultiHopQA, and HotpotQA, we evaluate acquisition, retention, and the joint use of evidence received at different updates. Under these schedules, Memini’s hybrid readout reduces mean answer-F1 forgetting by 50–72% relative to HippoRAG2 using its released incremental update path. It also improves cross-arrival answer F1 by 1.44–7.71 points over dense retrieval alone. Shuffling concept connections degrades retained answering even with the source archive unchanged, indicating that the organization of stored experience contributes beyond source availability. These results support source-grounded associative memory as a mechanism for system-level continual learning without adapting pretrained models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, associative memory, external memory, retrieval-augmented generation, multi-hop question answering, memory-augmented LLMs, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DJbUOpenReview ↗
Rating
3
Confidence
3(−1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Frozen-LLM external memory for a stream of unlabeled docs. each doc is an immutable “episode”, an LLM writer pulls out concepts and records concept-episode links + witnessed pairs. at query time they do a 1-hop walk: cue -> anchor episode that witnessed the pair -> bridge concept -> target episode. lots of hand scores (mention counts, title bonus, embedding focus etc). two readouts: assoc top5, or hybrid = dense top4 + 1 assoc slot. eval on HippoRAG2’s 1k MuSiQue/2Wiki/Hotpot sets, they split acquisition vs retention (4 cohort orders), also cross-arrival of supports, a shuffle control, writer/cue swaps, cost

Critical strengths
  • right problem. freezing weights doesnt magically solve CL if new/old evidence still fight over a 5-doc context. measuring acquisition / retention / BWT / forgetting / cross-arrival separately is way more useful than one end-of-corpus F1
  • the shuffle test is the part I actually care about. keep the archive, scramble dest concepts, assoc retention F1 drops a lot and hybrid still loses ~2-9 pts. so its not just “we stored the docs”. same answerer, same budget, same dense encoder, they report CIs, and they admit one-hop / no contradiction / ~6-12k scale
Critical weaknesses
  • the system people will cite is hybrid 4+1. so “beats HippoRAG2/Dense” is mostly dense + one extra slot. standalone assoc is worse than Dense on MuSiQue and Hotpot (Table 2). they are pretty honest in the body but the abstract 50-72% less forgetting vs HippoRAG2 shouldnt be read as a pure memory win
  • writer is GPT-5.4, HippoRAG2 OpenIE is 4o-mini. sec 5.5 mostly shows hybrid F1 is stable, not that HippoRAG2 stays worse with the same writer
  • method-wise its a pile of string match + log counts + title bonuses + one hop. fine as a system, but nothing is really learned (no router, no alias beyond exact string, no fact supersession). 4-hop MuSiQue is basically out of scope by design. solid systems paper, not a new memory algo imo
Questions for authors
  1. if you give HippoRAG2 GPT-5.4 (or you use 4o-mini writer in the main table) do forgetting / 2Wiki C@5 gaps still hold
  2. hybrid is always 4 dense. what’s retained F1 for 5-dense / 3+2 / 0+5 on each set. you point at app B.6, just put the numbers in the main text
  3. how often does that 1 extra assoc slot actually add a new support vs duplicate a dense hit
  4. identity = normalized string. collision rate on these wiki dumps? (Paris the city vs a person). does a cheap embedding merge change retention
  5. you assume facts never go stale. even a tiny contradiction split to show the fail mode would help
2 comments
AuthorsOpenReview ↗

Response to Reviewer DJbU (1/2)

We thank the reviewer for their careful reading and constructive suggestions, and for recognizing the evaluation protocol, shuffle control and systems contribution. We address the five questions below.

Memini adapts by accumulating source-grounded associations while pretrained parameters remain frozen. Our contribution is this memory organization and its evaluation of acquisition, retention and cross-arrival recombination under a fixed context budget.

1. Do forgetting and 2Wiki Complete@5 gaps persist with matched models?

Yes. The hybrid retains lower forgetting on all three datasets and higher 2Wiki Complete@5 when model strength is matched. We chose GPT-5.4 because concept identification and mention grouping shape the persistent memory available to later queries, with writing incurred once per passage (USD 0.0060–0.0067; Section 5.6). Table B19 examines writer sensitivity within Memini. We now extend this analysis to a direct comparison with HippoRAG2, using GPT-4o-mini for Memini’s writer and question cues alongside HippoRAG2’s GPT-4o-mini configuration. The answerer, five-passage budget and arrival schedules remain fixed.

DatasetHippoRAG2 F1 forgetting (points)Hybrid F1 forgetting (points)Hybrid − HippoRAG2, points [paired 95% CI]
MuSiQue7.354.10−3.25 [−4.46, −2.02]
2Wiki9.333.39−5.94 [−7.46, −4.48]
HotpotQA3.141.45−1.69 [−2.53, −0.86]

These are forgetting reductions of 44.2%, 63.7% and 53.8%, respectively, with all three paired intervals excluding zero. On 2Wiki, hybrid retained Complete@5 improves from 63.70% to 67.20%, a +3.50-point difference [1.33, 5.63]. Hybrid retained F1 also exceeds HippoRAG2 on MuSiQue by +2.57 points [0.59, 4.55] and on 2Wiki by +2.37 [0.47, 4.30]. HotpotQA remains close: +0.40 [−0.86, 1.65].

We now use this matched-model result for the abstract and introduction’s forgetting headline to address the model-strength concern directly.

The matched comparison therefore supports the hybrid’s retention benefits. Standalone associative answering is more writer-sensitive and falls below HippoRAG2 in this configuration. Section 5.5 (Table 6) presents the matched-model comparison in the main text; Appendix B.5 (Table B20) reports both readouts, including BWT. HippoRAG2 uses its released incremental path to incorporate arriving passages into its existing index, making it an appropriate native baseline for this stream. Matching models addresses writer strength within that implementation.

2. Dense/associative allocation

Both 4+1 and 3+2 have higher retained-F1 point estimates than five-Dense on all three datasets. Below, we report the requested allocations alongside our fixed 4+1 default. Scores are retained F1 (%), and allocations denote Dense + associative selections. Memini uses the main GPT-5.4 writer/cue configuration; the answerer, five-passage budget and evaluation protocol remain fixed.

DatasetDense 5+0Hybrid 4+1 (default)Hybrid 3+2Associative 0+5
MuSiQue45.1950.2250.2643.61
2Wiki59.8267.2370.3568.37
HotpotQA73.0173.9273.6168.56

The two hybrid allocations achieve nearly identical F1 on MuSiQue. A second associative slot improves 2Wiki’s F1 from 67.23 to 70.35, while 4+1 performs best on HotpotQA. Thus, the benefit of combining dense and associative retrieval persists across both allocations, with the preferred balance varying by dataset. As requested, we now include this compact comparison in Section 5.2 (Table 3), with the fuller allocation analysis in Appendix B.6.

3. What does the fifth slot contribute?

The associative slot recovers supporting evidence absent from Dense’s top five, and beneficial replacements outnumber harmful replacements on every dataset. It cannot duplicate the first four Dense passages. We compare its selection with Dense’s fifth passage to measure whether it adds useful evidence within the same budget. Counts below are out of 1,000 questions per dataset at final retention; rows overlap.

Fifth-slot outcomeMuSiQue2WikiHotpotQA
Support absent from Dense’s top five18234149
Replaces non-support with support17734049
Replaces support with non-support47714
Same passage as Dense’s fifth190105241

The useful recoveries often reach well below Dense’s cutoff. On 2Wiki, which has the most recoveries, 114 of the 341 recovered supports lie outside Dense’s top 100, including 78 absent from its saved top 200. Thus, the fifth slot supplies complementary supporting evidence, including passages far below the dense retrieval cutoff (Appendix B.6, Table B22, Figure B5).

AuthorsOpenReview ↗

Response to Reviewer DJbU (2/2)

4. Identity collisions and embedding merging

Our audit found collisions in 11/100 sampled shared names on MuSiQue and 8/100 each on 2Wiki and HotpotQA. A collision occurs when normalized-name matching joins different entities or meanings across passages.

We sampled 100 names shared by multiple passages in each dataset. Codex examined their 1,287 passage contexts and proposed SAME, COLLISION or UNCERTAIN labels with supporting explanations. One author then checked all 46 flagged or uncertain names and 60 randomly selected SAME cases. The final counts incorporate these author judgments; the remaining 194 names retain Codex labels. Appendix C.1 provides the full sampling and checking procedure, with results in Table C1.

These identity collisions do not necessarily cause retrieval failures. In a HotpotQA example, the passages titled Muppet Treasure Island and Sam Eagle use “The Muppets” for characters and a television series, respectively. Although these local concepts have different types, they share one normalized identity; their source passages remain separate. When asked which film was released earlier, One Magic Christmas or Muppet Treasure Island, both readouts retrieve both supporting film passages at the final checkpoint and correctly answer One Magic Christmas, using the years 1985 and 1996. Memini therefore recovers the required evidence and supports correct answering in this case despite the ambiguous identity.

Combining normalized names with the types of local concepts and passage-context similarity from a lightweight embedding model could refine identity matching. However, types may be inaccurate, and similarity may reflect relatedness rather than identity. Whether such refinements improve retrieval or retention remains an empirical question requiring a separate comparison.

5. Updating outdated facts

Our small fact-update diagnostic shows that associative Memini can recover updated evidence while exposing failures to use it correctly. The paper places fact supersession outside the main evaluated setting, which assumes earlier facts remain valid. To examine what happens when this assumption is relaxed, we constructed 20 fictional two-hop questions, generated and checked in Codex. Each requires a bridge passage identifying the relevant entity and a separate fact passage.

Matched unchanged/updated continuations each add 20 passages to the same 120-passage corpus while preserving older sources. Passage dates and query reference times identify which fact applies. We test GPT-5.4 and GPT-4o-mini for writing and cues. Both readouts, Dense and the oracle share the answerer and five-passage budget. The oracle receives the bridge, earlier fact, later fact and two distractors, testing answer use when the necessary evidence is available.

After updates, associative Memini retrieves both required supports in all 20 cases with either writer, compared with 11 for hybrid and 6 for Dense: a +70-point Complete@5 advantage over Dense (paired 95% CI [50, 90]). Yet its answering falls from 20/20 in the unchanged condition to 16/20 with GPT-5.4 and 14/20 with GPT-4o-mini, both above the hybrid’s 6/20 and Dense’s 4/20. Every remaining associative error is a stale answer despite complete retrieval. The oracle likewise falls from 20/20 to 16/20, even with the necessary evidence supplied.

The matched branches reveal additional difficulty in using changed facts, with question structure and corpus growth held fixed. Associative Memini recovers the required updated evidence without an added recency filter, deletion rule or supersession mechanism, while its remaining errors concern using that evidence correctly. These 20 constructed cases expose the requested failure mode while distinguishing successful evidence recovery from correct answer use (Appendix C.2, Table C2, Figure C1). The supplementary dataset ZIP provides the passages, questions, annotations and generation instructions for inspection.

Reviewer_rkdCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Memini, an external associative memory for continual learning from unlabeled textual observation streams while keeping all pretrained components frozen. Memini preserves each observation as an immutable source episode, extracts and deterministically checks salient local concepts, records concept–episode memberships and witnessed concept co-observations, and at query time performs bounded cue---anchor---bridge associative recall to retrieve complete source episodes, optionally combined with dense retrieval in a hybrid readout. The paper evaluates acquisition, retention, and cross-arrival recombination on streams derived from MuSiQue, 2WikiMultiHopQA, and HotpotQA against BM25, dense retrieval, and HippoRAG2. Experiments show that the hybrid readout reduces mean answer-F1 forgetting and improves cross-arrival answer F1 over dense retrieval, while shuffling learned concept connections substantially degrades retained answering.

Critical strengths
  1. The paper formulates a clear continual external-memory setting and separately evaluates acquisition, retention, and cross-arrival recombination under replay-free, question-independent updates, making the evaluation better aligned with memory growth than fixed-corpus QA.

  2. Memini preserves immutable source episodes while using witnessed concept co-observations to support traceable associative recall, and the controlled route-shuffling and cost/scalability analyses provide useful evidence that the learned organization contributes to retained answering.

Critical weaknesses
  1. The paper’s central claim---that witnessed concept-pair organization improves continual access---is not sufficiently isolated from simpler alternatives. Route shuffling shows that connections matter, but the paper does not report an ablation that removes pair observations while retaining the same episodes, concept-episode memberships, and dense retrieval, nor does it compare against equally adapted structured-memory baselines such as GraphRAG or LightRAG under the same five-episode budget.

  2. The continual-learning evaluation is narrow: it uses only four cyclic arrivals over three multi-hop QA datasets, assumes earlier information remains valid, and does not test fact updates or contradictions. Moreover, on MuSiQue sequential chains, the demonstrated advantage is mainly on parallel evidence branches; this limits external validity and makes broad continual-learning claims premature.

Questions for authors
  1. Can the authors ablate witnessed pair routes while retaining episodes, memberships, and dense retrieval, and include a degree-matched randomized-route control, to isolate gains from learned pair organization versus added retrieval diversity?

  2. Can the authors report results stratified by chain depth and arrival stage, and clarify whether conclusions extend beyond four arrivals and the valid-earlier-information assumption?

2 comments
AuthorsOpenReview ↗

Response to Reviewer rkdC (1/2)

We thank the reviewer for their careful reading and constructive suggestions, and for recognizing the evaluation setting and traceable associative recall. We address the mechanism controls, dependency depth, arrival stages and evaluation scope below.

1. Degree-matched control and the contribution of pair scoring

The shuffle already reported in the paper provides a degree-matched randomized-route control: each concept retains the same number of incoming and outgoing route records. We randomize the destination concepts while preserving these counts, counting separate witnesses separately. Source episodes, memberships, original witnesses and route attributes remain unchanged, as do the answerer, five-passage budget and hybrid’s four Dense selections. This tests whether the particular connections matter when source availability and the number of routes are held fixed (Section 5.4; Appendix B.4, Table B15).

Under this control, associative retained F1 falls by 31.29–56.68 points. The hybrid also loses 1.96–8.59 points, despite retaining the same four Dense passages. All six paired intervals exclude zero. These losses show that the particular connections contribute useful evidence even alongside dense retrieval: keeping the documents, route counts and retrieval budget unchanged does not preserve performance when those connections are disrupted.

The membership-only control produces smaller, mixed changes in Complete@5 and retained F1. To examine these scoring terms separately from the connection structure, we set pair quality to one and disable the boost from additional witnesses. Episodes, concepts, membership weights, question cues, semantic focus, eligible connections, strongest-contribution aggregation and the hybrid’s four Dense selections remain fixed.

We verified that this control can be reconstructed directly from concept–episode memberships, without consulting explicit pair records, with identical scores and rankings across all 48 evaluated checkpoints.

Full Memini has higher hybrid Complete@5 point estimates on MuSiQue (+0.80 points [−0.20, 1.80]) and HotpotQA (+0.10 [−0.80, 0.90]), while membership-only is higher on 2Wiki (full minus control: −0.90 [−1.90, 0.10]). Across both readouts, differences are at most 1.10 points, with all six paired 95% intervals including zero (Appendix B.4, Table B16).

DatasetReadoutFull Memini F1 (%)Membership-only F1 (%)Full − control, points [paired 95% CI]
MuSiQueAssociative43.6144.75−1.14 [−2.48, 0.21]
MuSiQueHybrid50.2249.91+0.31 [−0.67, 1.29]
2WikiAssociative68.3769.29−0.92 [−2.16, 0.26]
2WikiHybrid67.2367.59−0.36 [−1.10, 0.38]
HotpotQAAssociative68.5669.19−0.63 [−1.95, 0.63]
HotpotQAHybrid73.9274.23−0.31 [−0.95, 0.32]

Retained F1 changes by at most 0.36 points for hybrid and 1.14 points for associative recall, with all six intervals including zero (Appendix B.4, Table B17). Together with the degree-matched shuffle, these results support the usefulness of Memini’s concept-mediated retrieval, while pair-quality weighting and reinforcement have not shown a consistent benefit. The reinforcement examples in Appendix B.4 illustrate how additional witnesses can help individual questions or displace other required evidence.

We agree that matched comparisons with GraphRAG and LightRAG would broaden the evaluation. We have not run those comparisons under the same arrival protocol and five-passage budget.

2. Dependency depth and structure

The hybrid has the highest retained-F1 point estimate at every evaluated depth, including fully sequential four-step questions. To distinguish sequential dependencies from parallel evidence branches, we derive depth from the longest chain of dependent MuSiQue subquestions, rather than the number of supporting passages.

DepthQuestionsDense F1 (%)HippoRAG2 F1 (%)Hybrid F1 (%)
259148.7052.5755.72
330140.2541.3041.82
410839.7137.0143.48
AuthorsOpenReview ↗

Response to Reviewer rkdC (2/2)

At depth 2, the hybrid exceeds Dense by +7.02 points [4.40, 9.66] and HippoRAG2 by +3.16 [0.54, 5.81]. On fully sequential four-step questions, its advantage over HippoRAG2 is +6.47 [1.05, 11.89]. Its answering benefit is therefore not confined to parallel branches. The other direct comparisons remain uncertain. The hybrid also has higher retained-F1 point estimates than standalone associative recall at depths 3 and 4, where the associative scores are 29.75 and 30.29, respectively (Table B10).

Table B9 also shows that the hybrid has the highest point estimates across all four retrieval and answering metrics at acquisition and retention on these four-step chains. Complete recovery nevertheless remains difficult for every method: retained Complete@5 is 2.78% for the hybrid, versus 0.93% for Dense and 1.23% for HippoRAG2. The results therefore support improved answering on longer dependencies without establishing reliable recovery of entire chains. Appendix B.2 and Table B10 provide the depth breakdown, including both Memini readouts.

3. Arrival stages

The hybrid maintains higher F1 point estimates than Dense at every retention checkpoint on all three datasets. To examine how access to earlier evidence changes as memory grows, we follow the same first-arrival questions from acquisition at checkpoint 1 through retention at checkpoints 2–4. Each cyclic order contributes its 250 first-arrival questions, giving the same 1,000 questions at every checkpoint.

Paired 95% intervals support the hybrid’s gains over Dense at all three retention checkpoints on MuSiQue and 2Wiki, and at checkpoints 2 and 3 on HotpotQA. From acquisition to checkpoint 4, its observed F1 decline is smaller than either baseline’s on all three datasets. Although it starts below HippoRAG2, it finishes ahead on MuSiQue and 2Wiki by +2.78 points [0.70, 4.93] and +2.66 [0.64, 4.70], respectively.

The associative readout has the highest 2Wiki F1 at all three retention checkpoints, finishing +3.80 points above HippoRAG2 [1.29, 6.25]. On HotpotQA, hybrid F1 stays between 73.92 and 74.38, although its final differences from both baselines remain uncertain. Together, these trajectories show sustained answering on earlier questions as more passages compete for the same five-passage budget (Appendix B.2, Table B11).

4. Longer streams and changing facts

We agree that the temporal scope of the evaluation should be clear. Four arrivals define the evaluation schedule, rather than a limit built into Memini. Across the datasets, memory grows to 6,119–11,656 episodes while the answering budget remains five passages. The checkpoint results establish performance during this growth; retention over longer streams remains untested.

Our small fact-update diagnostic shows that associative Memini can recover updated evidence while exposing failures to use it correctly. The paper places fact supersession outside the main evaluated setting, which assumes earlier information remains valid. To examine what happens when this assumption is relaxed, we constructed 20 fictional two-hop questions, each requiring a bridge passage identifying the relevant entity and a separate fact passage. Matched unchanged/updated continuations add the same amount of material while preserving older sources. Passage dates and query reference times identify which fact applies.

We test GPT-5.4 and GPT-4o-mini for writing and cues. Both readouts, Dense and an oracle context share the answerer and five-passage budget. The oracle receives the bridge and both earlier and later facts, testing answer use when the necessary evidence is available.

After updates, associative Memini retrieves both required supports in all 20 cases with either writer, compared with 11 for hybrid and 6 for Dense: a +70-point Complete@5 advantage over Dense [50, 90]. Yet its answering falls from 20/20 in the unchanged condition to 16/20 with GPT-5.4 and 14/20 with GPT-4o-mini, both above the hybrid’s 6/20 and Dense’s 4/20. Every remaining associative error is a stale answer despite complete retrieval. The oracle likewise falls from 20/20 to 16/20, even with the necessary evidence supplied.

The matched branches reveal additional difficulty in using changed facts, with question structure and corpus growth held fixed. Associative Memini nevertheless recovers both required supports in every updated case without an added recency filter, deletion rule or supersession mechanism. Its remaining errors are stale answers despite complete retrieval, showing that access to updated evidence can succeed even when the answerer fails to use it correctly. These findings are limited to the 20 constructed cases (Appendix C.2, Table C2, Figure C1). The supplementary dataset ZIP provides the passages, questions, annotations and generation instructions for inspection.

Reviewer_YJ75OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

An external memory for LLM systems that stores each incoming document, uses an LLM to extract named concepts, and links documents that share concepts, recording which document “witnesses” each concept pair.

During query time, it follows one-hop routes (question concept → shared concept → other documents), either alone or as a fifth slot added to dense retrieval’s top four. On streams built from MuSiQue, 2Wiki, and HotpotQA, the hybrid shows 50–72% less answer forgetting than HippoRAG2 and modest gains over dense retrieval.

Critical strengths

A careful protocol: cyclic arrival orders, separate acquisition, retention, and cross-arrival metrics, and paired bootstrap intervals throughout.

Thorough analysis of complete cost and latency, including worked examples tracing exactly why retrieval succeeds or fails.

Critical weaknesses

The “continual learning” framing overstates the problem being solved. Earlier facts are assumed never to be superseded, so this is incremental indexing under growing distractor competition, not continual learning in the usual sense. The forgetting being measured is small (HippoRAG2 loses 3–9 F1, dense retrieval 1–4), and Memini’s reduction relative to dense isn’t significant on any dataset. The headline “50–72% vs. HippoRAG2” may largely reflect weaknesses in HippoRAG2’s released incremental update path rather than a strength of Memini. The comparison with HippoRAG2 is confounded by model strength. Memini’s writer is GPT-5.4; HippoRAG2’s extractor is GPT-4o-mini. Table B15 shows the effect: with a GPT-4o-mini writer, associative retained F1 on 2Wiki drops from 69.58 to 62.86, below HippoRAG2’s 65.04. The paper’s clearest win for the associative readout therefore depends on the stronger writer. A matched-model comparison is needed.

Questions for authors

Implementation details:

The gains appear to be dataset-specificm abd was there a separate developmetal data split?

Is there possible overfitting since good performance comes from 2Wiki’s templated bridge-comparison questions, where entity names match document titles exactly and exact-name concept linking does well.

How did you determine hand-set constants? How sensitive might they be?

2 comments
AuthorsOpenReview ↗

Response to Reviewer YJ75 (1/2)

We thank the reviewer for their careful reading and constructive suggestions, and for recognizing the evaluation protocol and cost analysis. We address the model comparison, development choices and scope below.

1. Matched-model comparison

The hybrid’s retention benefits persist with matched models. We chose GPT-5.4 because concept identification and mention grouping shape the memory available to later queries, while writing incurs a one-time cost per passage. Table B19 examines writer sensitivity within Memini. We now extend this analysis to a direct comparison with HippoRAG2, using GPT-4o-mini for Memini’s writer and question cues alongside HippoRAG2’s GPT-4o-mini configuration. The answerer, five-passage budget and arrival schedules remain fixed.

DatasetHippoRAG2 retained F1 (%)Hybrid retained F1 (%)Hybrid − HippoRAG2, points [paired 95% CI]
MuSiQue47.5050.07+2.57 [0.59, 4.55]
2Wiki65.0467.41+2.37 [0.47, 4.30]
HotpotQA72.9773.37+0.40 [−0.86, 1.65]

The retained-F1 advantage is supported on MuSiQue and 2Wiki; HotpotQA scores remain close. Hybrid F1 forgetting is also lower on every dataset, with all three paired 95% intervals excluding zero:

  • MuSiQue: 4.10 versus 7.35, a 44.2% reduction; difference −3.25 points [−4.46, −2.02].
  • 2Wiki: 3.39 versus 9.33, a 63.7% reduction; difference −5.94 [−7.46, −4.48].
  • HotpotQA: 1.45 versus 3.14, a 53.8% reduction; difference −1.69 [−2.53, −0.86].

Hybrid Complete@5 improves on 2Wiki from 63.70 to 67.20: +3.50 points [1.33, 5.63]. It also has a higher point estimate on MuSiQue, 45.90 versus 44.23 (+1.67 [−0.63, 3.97]), and remains close on HotpotQA, 92.00 versus 92.53 (−0.53 [−1.93, 0.83]).

The standalone readout is more writer-sensitive: its 2Wiki F1 is 62.86 with GPT-4o-mini, below HippoRAG2’s 65.04. However, Table B19 also shows that full GPT-5.4 is not uniquely required: GPT-5.4-mini achieves 69.61 versus 69.58, with GPT-4o-mini cues fixed. The matched comparison therefore supports the hybrid’s benefits while clarifying the standalone result’s dependence on writer choice. Section 5.5 (Table 6) presents the matched-model comparison in the main text; Appendix B.5 (Table B20) provides both readouts’ full comparisons, including BWT.

2. Development data and possible overlap

Development used small initial MuSiQue tests, and the retained-F1 advantage persists after excluding their 200 retrieval questions. The initial extraction pilot comprised 150 passages from the dataset’s 21,100-passage validation/dev pool, larger than the later 11,656-passage evaluation corpus: 100 randomly sampled passages and 50 selected to cover challenging extraction cases, such as acronyms, pronouns and dates.

Within these initial tests, we also used 200 questions to examine how to retrieve relevant evidence from the stored passages and concepts, holding memory fixed. We tested Personalized PageRank (PPR), which seeds activation from the question and propagates it through fixed graph transitions. Because this propagation could spread activation to question-irrelevant regions, we chose bounded, query-conditioned routes to keep evidence gathering focused within the small retrieval budget. The continual evaluation then tested acquisition and retention as memory grew, using this fixed configuration across MuSiQue, 2Wiki and HotpotQA without further tuning.

Although these initial retrieval tests differed from the continual evaluation, their questions occur in the final 1,000-question MuSiQue sample. We therefore excluded them as an additional check. On the remaining 800 questions, the hybrid retains a +3.18 F1 advantage over HippoRAG2 [0.90, 5.41]. The advantage thus persists beyond the questions used in the initial tests (Appendix B.7, Table B23).

AuthorsOpenReview ↗

Response to Reviewer YJ75 (2/2)

3. 2Wiki templates and exact-name matches

The hybrid’s benefit is not confined to bridge-comparison questions. Using the main configuration, we examined all four annotated question types. Excluding the 235 bridge-comparison questions leaves a hybrid retained-F1 advantage of +2.67 points [0.76, 4.62] on the other 765 questions. Its gains on compositional and comparison questions also have paired intervals above zero. Associative recall improves on inference questions by +9.11 points [1.70, 16.49] (Table B24).

To examine exact names separately, we grouped the same 1,000 questions by whether an extracted cue equals a full supporting title after Memini’s normalization. Both readouts improve retained F1 over HippoRAG2 on the 934 matched questions, with paired intervals above zero; differences in the 66-question unmatched group remain uncertain. For 771 matched questions, the matching name also appears in the same supporting passage’s body. Of the 66 without a title match, 62 have a cue appearing in a supporting body.

Together, these findings highlight exact names as useful entry points for Memini, with most title-matched names also occurring in the supporting passage’s body. A title match therefore does not mean that the name is available only through the title (Appendix B.7, Table B25).

4. Hand-set constants and sensitivity

On 2Wiki, the membership-title bonus improves complete-support recovery, while performance varies modestly across the tested weights. The design assumes that titles identify a passage’s central concepts. The membership bonus and pair-title multiplier therefore provide modest extra weighting, with defaults of 0.5 and 1.5; we do not claim these values are optimal.

We tested membership bonuses {0, 0.25, 0.5, 0.75} and pair-title multipliers {1.0, 1.25, 1.5, 1.75}, varying one weight at a time while keeping the remaining retrieval configuration fixed.

Complete@5 ranges from 79.10–80.70 for associative recall and 68.30–69.30 for hybrid, all numerically above HippoRAG2’s 63.70. Removing the membership bonus reduces Complete@5 by 1.20 points [0.50, 2.00] for associative recall and 0.70 [0.10, 1.30] for hybrid. Varying the pair-title multiplier changes Complete@5 by at most 0.10 points from the default. These results support the membership bonus’s contribution without suggesting that strong retrieval depends on a narrowly chosen default (Appendix B.7, Table B26).

5. Continual-learning scope and forgetting

We study continual learning at the system level, where persistent memory changes as observations arrive while pretrained components remain fixed. We agree that this framing should be tied clearly to the evaluated setting. To make that scope clear, our formulation in Section 2 specifies that earlier facts remain valid. Within this setting, preserving documents does not guarantee retaining usable knowledge: later arrivals can change whether earlier evidence is retrieved within the fixed five-passage budget. We therefore evaluate whether the system acquires new knowledge, retains previously supported answering capabilities and combines evidence received at different arrivals.

To clarify the forgetting comparison, the main configuration’s 50–72% reduction concerns the hybrid relative to HippoRAG2’s released incremental path. The matched-model results above strengthen this finding: forgetting remains 44–64% lower across all three datasets, with all three paired 95% intervals excluding zero (Section 5.5, Table 6; Appendix B.5, Table B20). To address model strength more directly, we now lead with this matched-model result in the abstract and introduction. Against Dense, the supported benefits concern retained F1 on MuSiQue and 2Wiki and cross-arrival F1 on all three datasets; forgetting differences remain uncertain, as reported in Section 5.2.

We chose HippoRAG2’s released incremental path because it incorporates new passages into its existing index, making it an appropriate native baseline for this stream. We recognize that the observed gap may also depend on this implementation; matching models addresses writer strength but does not separately isolate update behavior.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript and guide to changes

We have uploaded a revised manuscript and address each review in separate replies. To make the revisions easy to identify, new and substantively revised text appears in dark blue, with new tables and figures identified through their captions. The highlighting is for review and will be removed from the final manuscript.

The main changes are:

  • Matched-model comparison: Section 5.5, Table 6 presents the GPT-4o-mini comparison; Appendix B.5 provides the full results for both readouts.
  • Retrieval allocation and supporting evidence: Section 5.2, Table 3 compares Dense/associative allocations. Appendix B.6 examines the fifth slot’s contribution and where Dense ranked the recovered supports.
  • Mechanism ablation: Section 5.4 and Appendix B.4 add the membership-only control alongside the existing shuffle and reinforcement controls, with Complete@5 and retained F1 results in Tables B16 and B17.
  • Depth, arrival stages and development choices: Appendix B.2 adds dependency-depth results and acquisition-to-retention trajectories across checkpoints 1–4. Appendix B.7 documents the initial development tests, exclusion analysis, question-type and exact-name breakdowns, and title-weight sensitivity.
  • Identity collisions and changing facts: New Appendix C presents the shared-name audit and small fact-update diagnostic. The supplementary dataset ZIP contains the diagnostic’s passages, questions, annotations and generation instructions.

To address the model-strength concern directly, the abstract and introduction now lead with the matched GPT-4o-mini comparison, where hybrid F1 forgetting remains 44–64% lower across all three datasets.

The Discussion, reproducibility statement and AI-use statement have been updated accordingly. Our individual replies include the key findings and evidence directly, with references to these sections for fuller details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.