acceptodds
Under review as a conference paper at ICLR 2027

MemFuse: Multi-Source Memory Fusion from Fragmented Observations

Abstract

Long-term memory is essential for agents that operate across extended interactions, yet existing memory systems and benchmarks predominantly focus on single-source textual histories. In realistic settings, however, relevant information is often fragmented across applications and devices, as well as across users and time, requiring agents to integrate dispersed observations into coherent episodic memories while preserving their source provenance. To address these gaps, we introduce MemFuseBench, a benchmark for multi-source memory fusion. MemFuseBench is built with a Scene-to-Sensor pipeline that synthesizes controllable scenarios into source-tagged observations, evidence-grounded questions, and adversarial distractors. It enables systematic evaluation of temporal reasoning, cross-source evidence fusion, and robustness to noise. We further propose MemFuse, a structured memory system that preserves source-level evidence in event-layer atomic memory and organizes related atomic events into cluster-layer fused memory within a causal fusion graph. During retrieval, MemFuse retrieves and organizes related evidence fragments while maintaining traceability to original source events. Experiments on MemFuseBench show that MemFuse achieves the best overall performance among the evaluated memory systems under all three LLM settings and consistently improves performance on questions requiring cross-source evidence fusion.

Primary area:
datasets and benchmarks
Keywords:
Multi-Source Memory Fusion, Long-Term Memory, Episodic Memory, Memory Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HcgFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper studies multi-source memory fusion for agents whose evidence is fragmented across devices, users, and time.
  • It introduces MemFuseBench, a fully synthetic benchmark containing 7,823 events and 357 questions across six scenarios and six diagnostic categories. A Scene-to-Sensor pipeline constructs personas and causal storylines, projects them into source-specific observations, and generates evidence-grounded questions, answer checklists, and distractors, with reviewer-corrector validation.
  • The paper also proposes MemFuse, a training-free memory system combining immutable source-tagged atomic events, fused cluster memories, and membership, causal, and semantic graph relations. LLM agents organize incoming events and perform iterative constrained retrieval; the final answer is generated from 20 retrieved atomic events.
  • Across three LLM settings, MemFuse achieves the highest observed overall checklist coverage, although it does not consistently outperform the full-context reference. The ablations indicate that iterative retrieval and retrieval constraints account for the largest overall gains.
Critical strengths
  1. The benchmark targets a meaningful and useful memory problem. Complementary observations, source conflicts, and different user perspectives provide a focused setting for studying evidence integration beyond recalling facts from a conversational history. Explicit evidence-event identifiers and answer checklists make the task more auditable and potentially useful for diagnosing retrieval failures.
  2. The method preserves original evidence while using fused summaries as retrieval abstractions. This separation is a sensible design for maintaining provenance during memory aggregation. Experiments across three LLM settings show substantial improvements over naive RAG, and the component ablations and token-usage reporting help characterize both effectiveness and cost.
Critical weaknesses
  1. It looks like wo/ fusion doesn't drop much performance. Not sure if it is a useful mechanism, especially if the author uses muse as the main story.
  2. Unclear compute-performance tradeoff. A shared top-20 item limit does not equalize context length or internal LLM/search computation. Table 2 reports roughly 9-11 times more inference tokens for MemFuse than naive RAG. This matters for attributing the gains to memory organization and assessing the performance-cost tradeoff.
Questions for authors
  • Can you isolate the contribution of memory fusion and graph structure using a baseline with the models and budgets, but neither fused memories nor graph traversal? How does the comparison change under matched context-token and LLM-call/token budgets?
  • Could you explain why wo/ fusion doesn't have much impact on the method?
  • Please clarify the relationship to prior event-centric and episodic memory systems, including CompassMem [1], REMem [2], and Amory [3]. A more explicit comparison would help establish which contributions are specific to multi-source fusion.
  • Figure 4 is not easy to read because the caption doesn't give much explanation for these abbreviations.

[1] Yuyang Hu, Jiongnan Liu, Jiejun Tan, Yutao Zhu, and Zhicheng Dou. “Memory Matters More: Event-Centric Memory as a Logic Map for Agent Searching and Reasoning.” Findings of ACL 2026 [2] Yiheng Shu, Saisri Padmaja Jonnalagedda, Xiang Gao, Bernal Jiménez Gutiérrez, Weijian Qi, Kamalika Das, Huan Sun, and Yu Su. “REMem: Reasoning with Episodic Memory in Language Agents.” ICLR 2026 [3] Yue Zhou, Xiaobo Guo, Belhassen Bayar, and Srinivasan H. Sengamedu. “Amory: Building Coherent Narrative-Driven Agent Memory through Agentic Reasoning.” EACL 2026

5 comments
AuthorsOpenReview ↗

Response to W1/Q2: Why Removing Fused Memory Has a Small Effect

Thank you for raising this; we agree it needs a clear explanation.

In short. In the submission, the agentic controller masked the effect of fused memory. In the revised retrieval the structure as a whole has a significant effect, and fused grouping and graph propagation overlap, so removing either one alone costs less than removing both.

As described in the General Response, we revised MemFuse's retrieval after the controller-matched comparison: the iterative LLM controller is replaced by LLM-free, query-time aggregation over the fused nodes and the graph, with the same reader budget as naive RAG. This change is what makes the effect of the structure measurable. In the submitted version, removing fused memory changed Overall by only 0.0080 because the iterative controller could recover the same evidence through additional search rounds. In the revised retrieval no controller can compensate, and the effect becomes visible: on Gemini 3.1 Flash Lite, removing fused memory and the graph together lowers Overall from 0.3636 to 0.3151.

The two components overlap. Both connect observations of the same episode: fused nodes group them at write time, and the graph links related activities at query time. Fused grouping therefore adds 0.0179 when the graph is absent but only 0.0098 when it is present. Before the reader, fused grouping has the larger effect: removing it lowers gold-evidence recall@20 from 0.3565 to 0.3219, more than removing the graph (0.3345). Part of this retrieval gain does not carry over to the answer score at k=20, because related records from different devices often repeat the same fact.

Summary. We present fused memory and the graph as one structure whose joint contribution is significant, rather than claim a large independent effect for fused grouping, and we revise the paper's framing accordingly.

AuthorsOpenReview ↗

Response to Q1: Separating the Structure from the Retrieval Controller

Thank you for your positive assessment and for suggesting this precise baseline.

In short. We ran exactly the baseline you described: under the same iterative controller, the structure adds less than about two points. We then isolated the structure by letting it perform the aggregation without any controller; on Gemini, removing it lowers Overall by 0.0485.

With the same iterative controller, hybrid retrieval, temporal/source constraints, and reader, adding fused memory and the graph changes Overall by less than about two points. This confirms your reading: once an LLM controller iteratively searches, it performs most of the cross-source aggregation itself.

This result motivated the revised retrieval (see General Response). Rather than letting a controller compensate for the structure, we use the structure as the aggregation mechanism and remove all retrieval-time LLM calls. In this setting the contribution of the structure is isolated: on Gemini 3.1 Flash Lite, with an identical retriever, reader, and 20-event budget, removing fused memory and the graph lowers Overall from 0.3636 to 0.3151 and gold-evidence recall@20 from 0.3565 to 0.2970. Graph propagation accounts for most of this (0.0306); fused grouping adds 0.0098 on top of the graph. Under GPT-4.1 Mini and Qwen3-30B-A3B, MemFuse exceeds naive RAG by 0.0183 and 0.0093 at the same reader budget, but under Qwen it still trails Mem0 and EverMemOS at k=20.

Summary. The structure's distinct value is clear once it, rather than an LLM controller, performs the aggregation. The size of the gain varies with the reader and ingestion model, and we report the Qwen result as a limitation.

AuthorsOpenReview ↗

Response to W2/Q1: Matched Computation and the Performance-Cost Trade-off

Thank you for this insightful point about computation.

In short. The revised retrieval removes this confound: MemFuse and naive RAG now use about the same inference tokens and no retrieval-time LLM calls, and MemFuse improves over naive RAG under all three readers. Compared at matched inference cost, MemFuse matches or exceeds the strongest memory baseline in every setting.

We agree with you that a shared item count did not equalize computation in the original configuration. In the revised retrieval:

  • Matched reader context. MemFuse and naive RAG both pass at most 20 atomic events to the reader: 0.85 vs 0.84M, 0.77 vs 0.76M, and 0.90 vs 0.90M inference tokens over 357 questions (Gemini / GPT-4.1 Mini / Qwen3-30B-A3B).
  • Matched retrieval computation. MemFuse makes no LLM calls at retrieval time; like naive RAG, it computes one query embedding, plus lightweight score aggregation over the stored graph. Each question uses one LLM call (the reader).
  • Ingestion. All results reuse the memory snapshots of the submission, so ingestion cost is unchanged (29.73M–93.27M tokens, now reported in Table 2). This is a one-time offline cost that grows with the number of events, not the number of questions.

Under this matched budget, MemFuse improves over naive RAG by 0.0399 for Gemini, 0.0183 for GPT-4.1 Mini, and 0.0093 for Qwen3-30B-A3B. To address the performance-cost trade-off you raised, we also vary the reader budget (k = 20, 40, 80, 160, and 320 for all three readers) and compare with the other memory systems at matched inference cost (Figure 4 of the revision; tokens summed over 357 questions).

With larger budgets MemFuse also reaches the long-context reference, which reads the full event stream: 0.5373 vs 0.5223 for GPT-4.1 Mini at k=160 (4.91M tokens; long-context tokens are not available in this setting), 0.4733 vs 0.4424 for Qwen3-30B-A3B at k=160 (5.40M vs 40.83M tokens), and 0.5308 vs 0.5201 for Gemini at k=320 (10.50M vs 39.63M tokens). At k=320, MemFuse reaches 0.5606 under GPT-4.1 Mini (9.24M tokens) and 0.4774 under Qwen3-30B-A3B (10.32M tokens).

We do not attribute these budget gains to the fused structure: much of the improvement at larger k comes from passing more evidence to the reader, and the structure's measured contribution is the k=20 ablation reported above.

Summary. At matched reader tokens and retrieval computation, MemFuse improves over naive RAG under all three readers. Across budgets, it matches or exceeds the strongest memory baseline in each setting at lower or comparable inference cost. The revised Table 2 reports inference and ingestion tokens for every system, and its caption states the retrieval LLM calls.

Reviewer_ouHcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MemFuseBench for memory reasoning over fragmented multi-source observations. The benchmark contains 357 questions and 7,823 events across six synthetic scenarios. The paper also proposes MemFuse, a memory system with atomic events, fused memories, and a causal fusion graph. Experiments cover three LLM settings. MemFuse achieves the highest reported Overall score among the evaluated top-k retrieval and memory systems.

Critical strengths

1.It closes 62%–78% of the Fusion-category gap between naive RAG and long-context prompting. These results highlight the difficulty of recovering complementary evidence through standard nearest-neighbor retrieval.

2.The Scene-to-Sensor pipeline produces source-tagged observations, evidence-grounded questions, and adversarial distractors. Its six diagnostic categories support targeted analysis of different memory failures.

Critical weaknesses

1.Figure 4 shows an Overall decrease of only 0.0080 after removing fused memory. Removing agentic retrieval causes a much larger decrease of 0.1036. The paper does not include an agentic RAG baseline with the same query planning, filters, and search budget. The added value of the proposed memory structure therefore remains unclear.

2.The evaluation covers 357 questions from only six generated scenarios. No evaluation on an independent multi-source dataset is reported. The results therefore do not establish whether the improvements extend beyond the proposed generation pipeline.

3.The paper exhibits a mismatch between the main text and the appendix in how the evaluation protocol is described. While Section 5.1 specifies an input of 20 atomic events, Appendix C.2 indicates that fused summaries are included and counted as single entries. Consequently, the actual context used for comparison remains ambiguous and inadequately justified.

Questions for authors

1.The authors do not clearly explain how the six scenarios were utilized during prompt and parameter tuning, nor do they report results on independently constructed test scenarios to convincingly substantiate generalizability.

2.The authors fail to clarify the precise reader context evaluated in Table 2, introducing a direct contradiction between the claimed atomic-only protocol and the inclusion of fused summaries without reporting strict atomic-only benchmark results.

6 comments
AuthorsOpenReview ↗

Response to W1: Controller-Matched Baseline and the Value of the Structure

Thank you for your careful review and for this precise reading of Figure 4.

In short. We agree with your interpretation. The agentic baseline you requested shows that the structure adds less than about two points on top of the controller. With the revised retrieval, where the structure performs the aggregation itself, removing it on Gemini lowers Overall by 0.0485.

We ran the baseline you suggested (Agentic Atomic RAG: the same planner, filters, rounds, per-round top-20, hybrid search, and reader, with atomic events only). With the controller held fixed, adding fused memory and the graph changes Overall by less than about two points. The iterative controller already performs most of the cross-source aggregation, which explains the large effect of agentic retrieval in Figure 4.

We therefore revised retrieval so that the structure itself does the aggregation, with no LLM controller (see General Response). In this setting the structure can be tested without anything else compensating for it. On Gemini 3.1 Flash Lite, at the same 20-event budget and with an identical retriever and reader, removing fused memory and the graph lowers Overall from 0.3636 to 0.3151. Fused grouping on its own adds 0.0098 when the graph is present, which is not significant, and graph propagation accounts for 0.0306. The same ordering holds before any answer is generated: gold-evidence recall@20 drops from 0.3565 to 0.2970. Compared with naive RAG at the same reader budget, MemFuse improves Overall by 0.0399, 0.0183, and 0.0093 under Gemini, GPT-4.1 Mini, and Qwen3-30B-A3B.

Summary. The added value of the structure is small under an agentic controller and clear when the structure performs the aggregation itself, where most of it comes from graph propagation. Under Qwen3-30B-A3B the gain over naive RAG is small and MemFuse trails Mem0 and EverMemOS at k=20, which we report as a limitation.

AuthorsOpenReview ↗

Response to W2: Generalization Beyond Six Generated Scenarios

Thank you for raising this concern.

In short. We agree with you that six scenarios from one pipeline cannot establish generalization beyond it. Within the benchmark the gain is stable across scenarios, and we state the lack of an independent dataset as a limitation.

Within the benchmark, the results do not depend on a particular subset of scenarios: one retrieval configuration is used for all scenarios and all three LLM settings, and on Gemini the structure improves Overall on both halves of the benchmark (0.3821 vs 0.3151 on scenarios 1–3, 0.3433 vs 0.3151 on scenarios 4–6), although the gap is smaller on scenarios 4–6. For the ablation we report per-question paired bootstrap confidence intervals.

An independent multi-source dataset is not included in this revision.

Summary. The revision strengthens the statistical evidence within MemFuseBench, but it does not provide external validation. We state this as a limitation and do not claim generalization beyond the generation pipeline.

AuthorsOpenReview ↗

Response to W3/Q2: Reader Context Protocol

Thank you for catching this inconsistency; we appreciate your careful reading.

In short. You are correct: Appendix C.2 was accurate and Section 5.1 was not. The revised retrieval uses a strict atomic-only protocol: the reader receives at most 20 atomic events and no summaries, the same as naive RAG.

In the original agentic configuration, each search call ranked atomic events and fused nodes together. A selected fused node contributed its summary as one entry, followed by its member events, for at most three fused nodes, and all of these entries counted toward the 20-entry limit. Appendix C.2 describes this correctly; the phrase "20 atomic events" in Section 5.1 was inaccurate.

In the revised retrieval, fused nodes are used only to locate evidence, and each selected node is unpacked into its member atomic events. The reader receives at most k=20 source-tagged atomic events and no summaries; the budget is filled unless fewer visible events exist. This is the same format and about the same inference cost as naive RAG at k=20.

Summary. In the revised Table 2, MemFuse and naive RAG both pass only source-tagged atomic events to the reader, while Mem0, A-MEM, and EverMemOS return their native memory items as before. Table 2 now reports inference and ingestion tokens for every system. We have corrected Section 5.1 and updated Appendix C.2 to describe the revised protocol.

Reviewer_BbFvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies memory-based question answering when supporting information is fragmented across heterogeneous sources. It introduces MemFuseBench, a synthetic benchmark comprising 7,823 source-tagged textual observations and 357 questions across six scenarios. The proposed MemFuse system preserves atomic events and organizes related observations into fused memory nodes connected through membership, causal, and semantic links. An LLM-based agent maintains this structure during ingestion, while query-time retrieval combines hybrid search, graph expansion, and iterative evidence collection. Experiments with three backbone models compare MemFuse against existing memory systems, naive RAG, and a long-context baseline. The authors report the highest overall checklist coverage among the evaluated memory systems in all three settings, and provide component ablations and token-usage analyses.

Critical strengths

A meaningful problem formulation and a useful benchmark direction. The paper focuses on an important distinction between retrieving relevant memories and retrieving sufficient evidence to answer a question completely. MemFuseBench provides a controlled setting with heterogeneous, source-tagged observations, temporal dependencies, and conflicting information. This is a useful direction for evaluating memory systems beyond isolated fact retrieval and constitutes a substantive potential contribution. Consistent improvements of the complete system over naive retrieval. MemFuse improves overall checklist coverage over naive RAG by approximately 12.9–14.8 percentage points across three backbone models. The preservation of atomic events and explicit links to fused memories also provides an inspectable memory representation. Together, these results support the practical promise of the complete pipeline, even though the contribution of its individual architectural choices remains insufficiently established.

Critical weaknesses

However, the paper has the following weaknesses: The methodological novelty is limited. Designs that combine raw and higher-level memories, organize relationships among memories, and perform multi-round retrieval have already been extensively explored in existing systems. The paper lacks a controlled comparison that removes both the fusion layer and the relation graph while retaining the same query planning, retrieval constraints, and a comparable computational budget. The benchmark and evaluation do not adequately validate the cross-source fusion and evidence traceability capabilities emphasized in the paper. All experiments are conducted on six synthetic scenarios constructed by the authors, leaving insufficient evidence to establish that the observed improvements reflect these capabilities. In addition, the layout of Figure 1 is awkward. Despite the paper’s single-column format, the figure is arranged vertically, leaving substantial whitespace on both sides. This presentation issue detracts from the polish expected of a rigorous academic paper.

Questions for authors

What is MemFuse’s central novel contribution compared with existing hierarchical memory and agentic retrieval methods? Do the current experiments establish that the performance gains primarily come from the memory fusion mechanism? To what extent can MemFuseBench distinguish cross-source information fusion from general retrieval and question-answering capabilities? How do the authors validate this distinction? To what extent do MemFuse’s gains depend on the structure of the current synthetic data? What evidence supports its effectiveness in real-world settings with incomplete sources, erroneous records, or conflicting information?

5 comments
AuthorsOpenReview ↗

Response to W1/Q1: Scope of the Novelty Claims

Thank you for your thoughtful review and for recognizing the value of the problem formulation and the benchmark direction.

In short. You are right that the individual building blocks have precedent, and we do not claim them on their own. Our contributions are the multi-source problem, its benchmark, and a retrieval that uses write-time structure for query-time aggregation without LLM calls. We agree that keeping raw and summarized memory, linking memories, and multi-round retrieval each have precedent. Your comment helped us state our contribution more precisely:

  1. Problem. Existing memory systems and benchmarks mostly assume a single conversational or interaction history. We study observations of one episode that are split across devices, users, and time, where a system must retrieve complementary fragments while keeping the source of each.
  2. Benchmark. MemFuseBench provides source tags, gold evidence IDs, answer checklists, conflict events, and adversarial distractors. A Fusion question needs 15.2 distinct evidence events on average (Table 1), close to the 20-event budget, so the task directly tests whether a system collects a complete evidence set.
  3. Query-time aggregation over write-time structure, without retrieval-time LLM calls. Prior hierarchical and agentic systems either consolidate memories into summaries that replace raw records, or rely on an LLM controller to decide what to retrieve. MemFuse keeps every source-tagged atomic event immutable and uses fused nodes and graph links only as entry points: at query time, scores from matching atomic events are aggregated into nodes and propagated over the graph, and the selected nodes are unpacked back into their original atomic events. The aggregation therefore depends on the question, the reader sees only original source records, and retrieval costs about the same as embedding retrieval.

We measure the value of this design with a controlled ablation rather than list it as a contribution. On Gemini 3.1 Flash Lite, with an identical retriever, reader, and 20-event budget, removing fused memory and the graph lowers Overall from 0.3636 to 0.3151 and gold-evidence recall@20 from 0.3565 to 0.2970.

Summary. The novelty lies in the multi-source problem and benchmark and in how the structure is used at query time, not in hierarchical memory, graphs, or multi-round retrieval as such. Following your suggestion, we revise the contribution list to these three points and add the related-work comparison.

AuthorsOpenReview ↗

Response to W2/Q2: Isolating the Contribution of the Fusion Structure

Thank you for this important question, which directly shaped our new experiments.

In short. You are right that the original ablations did not establish this. Under a matched agentic controller, the structure adds less than about two points. With the revised retrieval, where the structure performs the aggregation and nothing else can compensate for it, removing it on Gemini lowers Overall by 0.0485.

We ran the comparison you requested under the agentic controller: with the same planner, constraints, rounds, and reader, adding fused memory and the graph changes Overall by less than about two points. An iterative LLM controller already does most of the cross-source aggregation, which is why agentic retrieval dominated Figure 4 of the previous submission.

This result led to the revised retrieval described in the General Response, where the structure itself performs the aggregation and the controller is removed. This also gives a cleaner test of your question, because nothing else in retrieval can compensate for the structure. On Gemini 3.1 Flash Lite, under an identical retriever, the same 20-event budget, and no retrieval-time LLM calls:

  • Removing both fused memory and the graph lowers Overall from 0.3636 to 0.3151 and Fusion from 0.2290 to 0.1885.
  • Graph propagation over the node-level links contributes most: removing it lowers Overall by 0.0306. Fused grouping adds 0.0098 when the graph is present, which is not significant, and 0.0179 when it is absent.
  • The same ordering appears before the reader: gold-evidence recall@20 is 0.3565, 0.3345, 0.3219, and 0.2970 for the full system, without the graph, without fused memory, and without both.

Under GPT-4.1 Mini and Qwen3-30B-A3B, MemFuse exceeds naive RAG by 0.0183 and 0.0093 at the same reader budget. Under Qwen it still trails Mem0 and EverMemOS at k=20; one possible factor is that the fused groups built by this ingestion model do not always match the evidence a question needs. We report this as a limitation.

Summary. When an agentic LLM controller is present, the structure adds little on top of it. When the structure itself performs query-time aggregation at a fixed budget, the Gemini ablation shows that graph propagation over the node-level links accounts for most of the gain over a structure-free retriever. The revised ablation section reports this result.

AuthorsOpenReview ↗

Response to W3/Q3: Retrieval-Only Evidence for Cross-Source Fusion and Traceability

Thank you for this helpful suggestion; it led us to add evaluations that separate retrieval from answering.

In short. We add metrics that do not involve the reader, and they show that the structure retrieves more of the required evidence before any answer is generated. Gains over naive RAG are consistent on the cross-device categories, and retrieval is a major bottleneck of the task. Every event the reader sees is an original source record.

We add three kinds of evidence.

  1. Retrieval-only metrics that do not involve the reader. We measure gold-evidence recall@20 and the checklist coverage of the retrieved context. Coverage counts a checklist item as covered when the context contains one of its annotated source events or a near-duplicate record of it (cosine ≥ 0.90), and uses neither the reader nor an LLM judge. On the Gemini snapshot, fused memory and the graph raise recall@20 from 0.2970 to 0.3565 and coverage from 0.384 to 0.432. These differences measure whether complementary evidence is retrieved, independently of the reader's QA ability.
  2. Gains appear on the cross-device categories. Relative to naive RAG (Table 2), MemFuse improves both cross-device categories in all three settings: Cross-source causality by 0.0755, 0.0318, and 0.0279, and Cross-source fusion by 0.0630, 0.0304, and 0.0206 (Gemini / GPT-4.1 Mini / Qwen3-30B-A3B). On Cross-user query the change is small or negative (−0.0078, −0.0172, +0.0071). This pattern is consistent with gains from cross-source retrieval rather than from general QA ability.
  3. Retrieval is a major bottleneck. With the same reader, the long-context reference exceeds naive RAG on Fusion by 0.2047–0.2706 across the three settings, so much of the gap reflects failure to retrieve complementary evidence. Even with all events visible, Fusion stays at 0.3737–0.4366, so the reader also contributes to the difficulty. In addition, questions answerable without the event stream are removed during construction (Appendix B, Stage 5).

Summary. These results indicate that MemFuseBench separates cross-source evidence collection from general QA ability, and that the gains of MemFuse already appear at the retrieval stage, before any answer is generated. Traceability is preserved by construction: the reader sees only original source-tagged atomic events and never fused summaries, so every event available to the answer carries its device tag and timestamp, and the retrieval trace records the selected event IDs.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Controller-Matched Comparison and Revised LLM-Free Retrieval

We thank all reviewers for their careful and constructive feedback, and we are glad that they find the problem meaningful and MemFuseBench auditable.

In short. The shared concern is whether the gains come from the fused memory and graph or from the iterative LLM retrieval controller. A controller-matched comparison confirmed that the controller does most of the aggregation. We therefore revised retrieval so that the structure itself performs this aggregation at query time, without LLM calls. At the same reader budget, MemFuse now improves over naive RAG under all three readers, and on Gemini 3.1 Flash Lite removing fused memory and the graph lowers Overall by 0.0485.

1. Controller-matched comparison. To answer this question, we ran Agentic Atomic RAG, which uses the same planner, hybrid retrieval, temporal/source constraints, iterative rounds, reader, and top-20 context as MemFuse, but stores atomic events only. Adding fused memory and the graph on top of this controller changes Overall by less than about two points, consistent with Figure 4 of the previous submission.

2. What this result tells us. Over several rounds, the controller collects complementary observations of the same episode from different devices and users. This is the aggregation that the fused nodes and graph are meant to provide, so with the controller present the structure is largely redundant, while the aggregation costs about ten times the inference tokens of naive RAG. The aggregation cannot simply move to write time either: fused nodes are built before any question is known, so they cannot decide on their own which evidence a question needs.

3. Revised retrieval. We therefore keep the aggregation at query time but let the structure perform it instead of an LLM:

  • Atomic events are ranked by RRF of dense (BGE-M3) and BM25 rankings.
  • Each fused node is scored by RRF over its summary similarity, its best member's similarity, and a member vote from top-ranked events; events outside fused nodes form single-event nodes.
  • Node scores are propagated one step over links that connect causally related activities and activities close in time for the same or co-present users.
  • The context takes the top five atomic events, then nodes in score order unpacked into member events, up to k=20.

The reader sees only source-tagged atomic events, never fused summaries. Retrieval uses only the query embedding, so it costs about as much as naive RAG; the memory snapshots and ingestion cost are unchanged; and one configuration is used for all scenarios and readers (see our response to Reviewer ouHc, Q1). Because no controller can compensate for the structure, its contribution can now be measured directly.

4. What the structure contributes. On Gemini, with an identical retriever and budget, removing fused memory and the graph lowers Overall from 0.3636 to 0.3151, and graph propagation alone accounts for 0.0306. Compared with naive RAG at the same reader budget, MemFuse improves under all three readers, most for Gemini and least for Qwen3-30B-A3B, where it still trails Mem0 and EverMemOS at k=20. With larger reader budgets, MemFuse matches or exceeds the strongest memory baseline in each setting at lower or comparable cost (response to Reviewer HcgF, W2).

Revisions (marked in blue). Table 2 is updated with the revised retrieval and with inference and ingestion tokens for all systems; Section 4.4, the ablations, and the reader-context description are rewritten; the appendix adds retrieval-only metrics, per-half results, the tuning protocol, and the related-work comparison requested by Reviewer HcgF. We no longer claim that MemFuse is best in every setting, and we state its limitations explicitly.

Summary. The structure adds little on top of an agentic controller, because the controller already performs the aggregation. When the structure performs this aggregation itself, without LLM calls and at the same budget, MemFuse improves over naive RAG under all three readers, and the Gemini ablation attributes most of the gain to graph propagation. We thank the reviewers again for comments that made the analysis more rigorous.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.