- Rating
- 3
- Confidence
- 3(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Frozen-LLM external memory for a stream of unlabeled docs. each doc is an immutable “episode”, an LLM writer pulls out concepts and records concept-episode links + witnessed pairs. at query time they do a 1-hop walk: cue -> anchor episode that witnessed the pair -> bridge concept -> target episode. lots of hand scores (mention counts, title bonus, embedding focus etc). two readouts: assoc top5, or hybrid = dense top4 + 1 assoc slot. eval on HippoRAG2’s 1k MuSiQue/2Wiki/Hotpot sets, they split acquisition vs retention (4 cohort orders), also cross-arrival of supports, a shuffle control, writer/cue swaps, cost
Critical strengths
- right problem. freezing weights doesnt magically solve CL if new/old evidence still fight over a 5-doc context. measuring acquisition / retention / BWT / forgetting / cross-arrival separately is way more useful than one end-of-corpus F1
- the shuffle test is the part I actually care about. keep the archive, scramble dest concepts, assoc retention F1 drops a lot and hybrid still loses ~2-9 pts. so its not just “we stored the docs”. same answerer, same budget, same dense encoder, they report CIs, and they admit one-hop / no contradiction / ~6-12k scale
Critical weaknesses
- the system people will cite is hybrid 4+1. so “beats HippoRAG2/Dense” is mostly dense + one extra slot. standalone assoc is worse than Dense on MuSiQue and Hotpot (Table 2). they are pretty honest in the body but the abstract 50-72% less forgetting vs HippoRAG2 shouldnt be read as a pure memory win
- writer is GPT-5.4, HippoRAG2 OpenIE is 4o-mini. sec 5.5 mostly shows hybrid F1 is stable, not that HippoRAG2 stays worse with the same writer
- method-wise its a pile of string match + log counts + title bonuses + one hop. fine as a system, but nothing is really learned (no router, no alias beyond exact string, no fact supersession). 4-hop MuSiQue is basically out of scope by design. solid systems paper, not a new memory algo imo
Questions for authors
- if you give HippoRAG2 GPT-5.4 (or you use 4o-mini writer in the main table) do forgetting / 2Wiki C@5 gaps still hold
- hybrid is always 4 dense. what’s retained F1 for 5-dense / 3+2 / 0+5 on each set. you point at app B.6, just put the numbers in the main text
- how often does that 1 extra assoc slot actually add a new support vs duplicate a dense hit
- identity = normalized string. collision rate on these wiki dumps? (Paris the city vs a person). does a cheap embedding merge change retention
- you assume facts never go stale. even a tiny contradiction split to show the fail mode would help
2 comments
Response to Reviewer DJbU (2/2)
4. Identity collisions and embedding merging
Our audit found collisions in 11/100 sampled shared names on MuSiQue and 8/100 each on 2Wiki and HotpotQA. A collision occurs when normalized-name matching joins different entities or meanings across passages.
We sampled 100 names shared by multiple passages in each dataset. Codex examined their 1,287 passage contexts and proposed SAME, COLLISION or UNCERTAIN labels with supporting explanations. One author then checked all 46 flagged or uncertain names and 60 randomly selected SAME cases. The final counts incorporate these author judgments; the remaining 194 names retain Codex labels. Appendix C.1 provides the full sampling and checking procedure, with results in Table C1.
These identity collisions do not necessarily cause retrieval failures. In a HotpotQA example, the passages titled Muppet Treasure Island and Sam Eagle use “The Muppets” for characters and a television series, respectively. Although these local concepts have different types, they share one normalized identity; their source passages remain separate. When asked which film was released earlier, One Magic Christmas or Muppet Treasure Island, both readouts retrieve both supporting film passages at the final checkpoint and correctly answer One Magic Christmas, using the years 1985 and 1996. Memini therefore recovers the required evidence and supports correct answering in this case despite the ambiguous identity.
Combining normalized names with the types of local concepts and passage-context similarity from a lightweight embedding model could refine identity matching. However, types may be inaccurate, and similarity may reflect relatedness rather than identity. Whether such refinements improve retrieval or retention remains an empirical question requiring a separate comparison.
5. Updating outdated facts
Our small fact-update diagnostic shows that associative Memini can recover updated evidence while exposing failures to use it correctly. The paper places fact supersession outside the main evaluated setting, which assumes earlier facts remain valid. To examine what happens when this assumption is relaxed, we constructed 20 fictional two-hop questions, generated and checked in Codex. Each requires a bridge passage identifying the relevant entity and a separate fact passage.
Matched unchanged/updated continuations each add 20 passages to the same 120-passage corpus while preserving older sources. Passage dates and query reference times identify which fact applies. We test GPT-5.4 and GPT-4o-mini for writing and cues. Both readouts, Dense and the oracle share the answerer and five-passage budget. The oracle receives the bridge, earlier fact, later fact and two distractors, testing answer use when the necessary evidence is available.
After updates, associative Memini retrieves both required supports in all 20 cases with either writer, compared with 11 for hybrid and 6 for Dense: a +70-point Complete@5 advantage over Dense (paired 95% CI [50, 90]). Yet its answering falls from 20/20 in the unchanged condition to 16/20 with GPT-5.4 and 14/20 with GPT-4o-mini, both above the hybrid’s 6/20 and Dense’s 4/20. Every remaining associative error is a stale answer despite complete retrieval. The oracle likewise falls from 20/20 to 16/20, even with the necessary evidence supplied.
The matched branches reveal additional difficulty in using changed facts, with question structure and corpus growth held fixed. Associative Memini recovers the required updated evidence without an added recency filter, deletion rule or supersession mechanism, while its remaining errors concern using that evidence correctly. These 20 constructed cases expose the requested failure mode while distinguishing successful evidence recovery from correct answer use (Appendix C.2, Table C2, Figure C1). The supplementary dataset ZIP provides the passages, questions, annotations and generation instructions for inspection.
Response to Reviewer DJbU (1/2)
We thank the reviewer for their careful reading and constructive suggestions, and for recognizing the evaluation protocol, shuffle control and systems contribution. We address the five questions below.
Memini adapts by accumulating source-grounded associations while pretrained parameters remain frozen. Our contribution is this memory organization and its evaluation of acquisition, retention and cross-arrival recombination under a fixed context budget.
1. Do forgetting and 2Wiki Complete@5 gaps persist with matched models?
Yes. The hybrid retains lower forgetting on all three datasets and higher 2Wiki Complete@5 when model strength is matched. We chose GPT-5.4 because concept identification and mention grouping shape the persistent memory available to later queries, with writing incurred once per passage (USD 0.0060–0.0067; Section 5.6). Table B19 examines writer sensitivity within Memini. We now extend this analysis to a direct comparison with HippoRAG2, using GPT-4o-mini for Memini’s writer and question cues alongside HippoRAG2’s GPT-4o-mini configuration. The answerer, five-passage budget and arrival schedules remain fixed.
These are forgetting reductions of 44.2%, 63.7% and 53.8%, respectively, with all three paired intervals excluding zero. On 2Wiki, hybrid retained Complete@5 improves from 63.70% to 67.20%, a +3.50-point difference [1.33, 5.63]. Hybrid retained F1 also exceeds HippoRAG2 on MuSiQue by +2.57 points [0.59, 4.55] and on 2Wiki by +2.37 [0.47, 4.30]. HotpotQA remains close: +0.40 [−0.86, 1.65].
We now use this matched-model result for the abstract and introduction’s forgetting headline to address the model-strength concern directly.
The matched comparison therefore supports the hybrid’s retention benefits. Standalone associative answering is more writer-sensitive and falls below HippoRAG2 in this configuration. Section 5.5 (Table 6) presents the matched-model comparison in the main text; Appendix B.5 (Table B20) reports both readouts, including BWT. HippoRAG2 uses its released incremental path to incorporate arriving passages into its existing index, making it an appropriate native baseline for this stream. Matching models addresses writer strength within that implementation.
2. Dense/associative allocation
Both 4+1 and 3+2 have higher retained-F1 point estimates than five-Dense on all three datasets. Below, we report the requested allocations alongside our fixed 4+1 default. Scores are retained F1 (%), and allocations denote Dense + associative selections. Memini uses the main GPT-5.4 writer/cue configuration; the answerer, five-passage budget and evaluation protocol remain fixed.
The two hybrid allocations achieve nearly identical F1 on MuSiQue. A second associative slot improves 2Wiki’s F1 from 67.23 to 70.35, while 4+1 performs best on HotpotQA. Thus, the benefit of combining dense and associative retrieval persists across both allocations, with the preferred balance varying by dataset. As requested, we now include this compact comparison in Section 5.2 (Table 3), with the fuller allocation analysis in Appendix B.6.
3. What does the fifth slot contribute?
The associative slot recovers supporting evidence absent from Dense’s top five, and beneficial replacements outnumber harmful replacements on every dataset. It cannot duplicate the first four Dense passages. We compare its selection with Dense’s fifth passage to measure whether it adds useful evidence within the same budget. Counts below are out of 1,000 questions per dataset at final retention; rows overlap.
The useful recoveries often reach well below Dense’s cutoff. On 2Wiki, which has the most recoveries, 114 of the 341 recovered supports lie outside Dense’s top 100, including 78 absent from its saved top 200. Thus, the fifth slot supplies complementary supporting evidence, including passages far below the dense retrieval cutoff (Appendix B.6, Table B22, Figure B5).