acceptodds
Under review as a conference paper at ICLR 2027

KMMG-Agent: A Knowledge–Memory Multi-Graph Agent for Multi-hop Egocentric Question Answering

Abstract

Large pretrained language models can store vast amounts of factual knowledge and leverage it to answer user queries. However, they remain limited when handling information beyond their knowledge boundaries. Equipping agents with external knowledge interaction and memory mechanisms has proven to be an effective strategy, enabling them to dynamically acquire explicit knowledge and incorporate it into the reasoning process, thereby improving answer accuracy. Nevertheless, existing memory retrieval and information matching methods often overlook temporal information in memory and lack effective modeling of the connections between dynamic memories and large-scale external knowledge. To address these limitations, we propose the Knowledge Memory Multi-Graph Agent (KMMG-Agent), which preserves repeated memory events as distinct timestamped edges and combines query-specific temporal filtering with intermediate-entity-based chained retrieval to support two-hop question answering over personal memories and Wikipedia knowledge. To systematically evaluate the effectiveness of the proposed framework across different cross-source reasoning scenarios, we further construct a multi-hop question-answering benchmark that integrates memory and external knowledge and covers four distinct reasoning paths. Experimental results show that KMMG-Agent outperforms existing baselines on all three memory-related tasks, achieving approximately 11.8%-33.2% improvements in Exact Match (EM) accuracy. These results demonstrate the effectiveness of KMMG-Agent in temporal memory retrieval and cross-source multi-hop reasoning.

Primary area:
datasets and benchmarks
Keywords:
Memory-Augmented Agents, Multi-Hop Question Answering, Temporal Memory Retrieval, Cross-Source Reasoning, Knowledge Graphs, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hxQxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work tackles the challenge of multi-hop question answering requiring integration of personal temporal memory data from first-person video captions and static external knowledge from Wikipedia. The authors propose KMMG-Agent, a framework that represents memories as a multi-graph with timestamped edges to preserve repeated events at different times, combining query-specific temporal filtering with intermediate entity-based chained retrieval. They also introduce Mem-WikiQA, a new benchmark with 2,000 two-hop questions covering four evidence-source paths (Memory-to-Memory, Memory-to-Wiki, Wiki-to-Memory, Wiki-to-Wiki). Experimental results show KMMG-Agent outperforms existing baselines by 11.8%-33.2% in Exact Match accuracy on memory-related tasks.

Critical strengths

Critical Strengths

1.The timestamped edge representation for temporal memories is an interesting idea that could potentially be applied in other memory modeling contexts.

2.The Mem-WikiQA benchmark addresses a relevant problem space of cross-source reasoning.

Critical weaknesses

Critical Weaknesses

1.Severe limited reasoning scope: The benchmark is artificially restricted to only two-hop reasoning, which is far less complex than real-world multi-hop QA scenarios. This severely limits the practical impact and generalizability of the work.

2.Unaddressed entity alignment critical flaw: The paper completely ignores entity alignment errors between memory and Wikipedia sources, which is a fundamental challenge in cross-source reasoning systems. Without addressing this, the proposed approach cannot be deployed in real-world applications.

3.Insufficient experimental validation: The experiments only compare with a limited set of baselines and do not include ablation studies that properly isolate the contribution of each component. The reported improvements are not convincingly demonstrated to be due to the proposed multi-graph design rather than other confounding factors.

4.Lack of novelty: The core ideas of multi-graph memory representation and temporal filtering have been explored in prior work, with only incremental modifications presented in this paper.

Questions for authors

None, as the paper has fundamental limitations that cannot be addressed with minor revisions.

1 comment

Response to Reviewer

We thank the reviewer for the constructive comments.

Comment 1. Limited reasoning depth of the benchmark

We thank the reviewer for this important comment. We acknowledge that real-world multi-hop QA often involves more complex reasoning processes than two-hop scenarios, requiring the integration of information across multiple entities, relations, and sources.

Our focus on two-hop reasoning is an intentional choice to validate the effectiveness of our multi-graph memory representation and temporal filtering mechanism in a controlled setting. This setting provides a manageable yet representative scenario for evaluating cross-source information integration, relational reasoning, and temporal-aware retrieval, while allowing us to better isolate the contributions of our proposed mechanisms.

We emphasize that two-hop reasoning reflects the scope of our current evaluation rather than an inherent limitation of the framework. Extending the evaluation to more complex scenarios involving three or more hops, as well as longer reasoning chains and more diverse temporal dependencies, is an important direction for future work.

Comment 2. Entity Alignment Between Memory and Wikipedia Sources

We appreciate the reviewer's valuable comment regarding entity alignment between memory and Wikipedia sources. We agree that entity matching across heterogeneous information sources is an important challenge in cross-source reasoning systems. In our approach, retrieval is performed through two complementary components: semantic retrieval and entity retrieval. Specifically, semantic retrieval identifies relevant information based on semantic similarity, enabling the system to retrieve contextually related content even when different sources use different expressions. Meanwhile, entity retrieval focuses on identifying and retrieving relevant entities, helping establish entity-level correspondences across different information sources. By combining these two retrieval mechanisms, our approach considers both semantic relevance and entity-level information, rather than relying solely on exact entity name matching.

Comment 3. Additional Ablation Studies

1. Ablation on Retrieval Mechanisms.

First, we conducted experiments with three retrieval configurations: Entity-only Retrieval, Semantic-only Retrieval, and Hybrid Retrieval (combining entity-based and semantic retrieval). These experiments allow us to examine the individual contributions of entity-based and semantic-based retrieval, as well as the effectiveness of combining these complementary mechanisms.

Table 1. Exact-match accuracy (%) for second-hop retrieval strategies under a matched evidence budget.

Retrieval StrategyMemory-to-MemoryMemory-to-WikiWiki-to-Memory
Entity-only46.861.633.2
Semantic-only50.469.248.0
Hybrid (Ours)53.870.449.4

2. Ablation on Evidence Sources.

Second, to further investigate the contribution of evidence from different sources, we conducted additional experiments by removing evidence from individual sources (i.e., personal memory or Wikipedia) and removing evidence from both sources. These experiments allow us to assess the importance of each evidence source and evaluate the impact of integrating cross-source evidence on overall performance.

Table 2. Exact-match accuracy (%) under different evidence availability conditions.

TaskBoth SourcesMemory OnlyWikipedia OnlyNo External Evidence
Memory-to-Wiki70.436.240.47.6
Wiki-to-Memory49.422.60.00.0

Overall, these additional experiments provide a more systematic evaluation of the proposed retrieval mechanisms and evidence integration strategy, offering further empirical support for the design choices underlying our framework.

Comment 4. Methodological Contributions and Novelty

We sincerely thank the reviewer for the valuable comments regarding the novelty of our work. Our contribution lies not in proposing individual techniques in isolation, but in integrating them into a unified framework for cross-source reasoning.

Specifically, our framework focuses on connecting heterogeneous information sources, particularly memory and Wikipedia, through three complementary mechanisms: (1) Multi-graph memory representation to organize and connect information from different sources; (2) Entity-based and semantic-based retrieval to identify relevant information by considering both entity associations and semantic relevance; and (3) Temporal filtering to select relevant evidence based on temporal information.

Together, these mechanisms provide an integrated approach to organizing, retrieving, and utilizing heterogeneous evidence for cross-source reasoning.

Reviewer_oDimOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Mem-WikiQA, a 2,000-question benchmark with four balanced two-hop paths connecting EgoLife caption-derived memories and Wikipedia facts. KMMG-Agent stores repeated events as timestamped edges, decomposes a question into two subquestions, filters memory facts by time, and combines entity-neighborhood and semantic retrieval at the second hop. Experiments with Qwen3-32B and GPT-5.4 report substantial gains over the listed baselines on the three memory-related tasks. The ablations show that temporal filtering accounts for much of this improvement; the additional advantage over time-filtered text retrieval is smaller and varies by task and backbone.

Critical strengths
  1. Mem-WikiQA provides a useful benchmark for connecting personal memories with external knowledge. Its 2,000 questions are evenly divided among four evidence-source paths, including both directions of memory-Wikipedia interaction. I see this as a central contribution: it makes the source dependency explicit and provides a controlled setting for examining where cross-source retrieval succeeds or fails. Using timestamped EgoLife captions, checking generated examples against the original evidence, and varying the temporal expressions gives the benchmark a concrete connection to personal-assistant tasks. This is a worthwhile extension of evaluating memory recall and knowledge retrieval separately.

  2. KMMG-Agent offers a coherent approach to retrieving temporally relevant evidence across the two sources. Retaining repeated interactions as separate timestamped events preserves information that could be lost through aggregation, while the predicted intermediate entity connects the two retrieval steps. The second-hop retriever combines entity connectivity with semantic relevance, providing a practical way to retrieve linked facts without relying entirely on exact entity matching. The substantial gains over the listed baselines on all three memory-related tasks, together with the temporal-filtering ablations across two backbones, support the usefulness of this overall design in the proposed setting.

Critical weaknesses
  1. The experiments establish the value of temporal filtering more clearly than the additional value of the multigraph and hybrid retrieval. In Table 3, the full method improves over time-filtered text retrieval by 2.8-3.4 percentage points on Memory-to-Memory. On the cross-source tasks, however, it is 0.4-0.6 points lower with Qwen3-32B and 1.2-1.4 points higher with GPT-5.4. The paper acknowledges this mixed result, but the comparison does not clearly separate the effects of event representation and the two retrieval branches. Since the methodological contribution centers on their combination, a more controlled comparison would help establish the specific advance over a simpler time-aware retriever.

  2. The evaluation leaves the reliability of the intended two-hop evidence chain insufficiently examined. The paper reports final-answer EM, but does not directly test how often both sources are needed for the cross-source questions; some general Wikipedia links could also be supplied by the backbone’s existing knowledge. In addition, Section 4.2 describes a single intermediate entity, whereas Appendix C.2 explicitly permits multiple answers. The handling of these cases, and whether the broader time intervals preserve a unique answer, is not explained. These gaps matter because a correct final answer alone does not show that the intended chain was followed, and ambiguous intermediate outputs can change the second-hop evidence.

Questions for authors
  1. I would like to understand which part of the proposed combination accounts for the improvement beyond temporal filtering. In Table 3, the full method is close to Text Retrieval w/ Time on the cross-source tasks, with different outcomes for the two backbones. Could you clarify which components differ between these rows and provide a focused comparison of entity-only, semantic-only and hybrid retrieval using the same total evidence budget? This would help me assess the contribution of the graph-guided retrieval.

  2. I find the benchmark’s cross-source setting interesting, but I would like to know how consistently the questions require both evidence sources. For example, the charger-to-battery connection in Figure 1 may already be familiar to the backbone. Did the manual screening check for such shortcuts? A small analysis that withholds Wikipedia or memory evidence in turn, together with a few inspected reasoning chains, would help distinguish successful cross-source retrieval from answers obtained using only part of the intended evidence.

  3. I noticed a difference between the single intermediate entity in Section 4.2 and the multiple-entity output allowed by Appendix C.2. Could you explain how these outputs are used to form the second-hop query and select the final answer? I would also appreciate clarification on whether answer uniqueness was rechecked after broadening the time expressions. These details would help me understand when multiple entities are valid answers and how such cases are treated by the EM evaluation.

2 comments
AuthorsOpenReview ↗

Retrieval Ablations, Cross-Source Evidence, and Multi-Entity Evaluation

Response to Reviewer

We thank the reviewer for the constructive comments. We clarify retrieval design, report controlled ablations, and explain multiple-entity evaluation.

Comment 1. Contribution beyond temporal filtering

Both methods(Text Retrieval w/ Time & KMMG) apply temporal filtering but differ in second-hop evidence selection. Text Retrieval w/ Time ranks time-eligible facts by semantic similarity without requiring a link to the predicted first-hop entity. Ours combines two parallel branches: (i) an entity-neighborhood branch (top 10) that semantically ranks facts whose head or tail matches the predicted entity; and (ii) a global semantic branch (top 20) that ranks all time-eligible facts without entity matching. Both branches share temporal constraints, and their evidence is combined for answer generation.

To isolate entity-neighborhood effects, we compared Entity-only, Semantic-only, and Hybrid retrieval with matched facts and total evidence budgets. For each question and backbone, we fixed the first-hop entity, instantiated second-hop query, temporal constraints, generation prompt, and decoding settings.

Table 1. Exact-match accuracy (%) for second-hop retrieval strategies under a matched evidence budget.

Retrieval strategyMemory-to-MemoryMemory-to-WikiWiki-to-Memory
Entity-only46.861.633.2
Semantic-only50.469.248.0
Hybrid (ours)53.870.449.4

Comment 2. Whether cross-source questions require both evidence sources

To probe knowledge shortcuts, we evaluated the same Memory-to-Wiki and Wiki-to-Memory questions under four conditions: Both Sources, Memory Only, Wikipedia Only, and No External Evidence. All four runs started independently from the original question. In particular, single-source and no-evidence runs did not reuse the dual-source run's intermediate entity, reformulated second-hop query, or retrieved evidence. Across conditions, the model could answer using the question, its available evidence, and its parametric knowledge, following the same answer-generation rules.

Table 2. Exact-match accuracy (%) with different evidence sources available.

TaskBoth sourcesMemory onlyWikipedia onlyNo external evidence
Memory-to-Wiki70.436.240.47.6
Wiki-to-Memory49.422.60.00.0

For each question, all four runs started independently from the same original question. In the single-source and no-external-evidence conditions, we did not reuse the intermediate entity identified in the dual-source run, the second-hop query reformulated using that entity, or the retrieval results from either hop. This prevented information from a withheld source from reaching the model indirectly through these intermediate outputs. All four conditions followed the same answering rule. The model could draw on the question itself, any evidence available in that condition, and its parametric knowledge. This design allowed us to compare, for each question, whether the model could still answer correctly when we withheld personal memory, Wikipedia, or both. Table 2 reports the results. On both Memory-to-Wiki and Wiki-to-Memory, exact match (EM) scores were substantially higher with dual-source retrieval than with no external evidence. These results suggest that the model struggles to answer these questions using its parametric knowledge alone and that jointly retrieving evidence from personal memory and Wikipedia improves answer accuracy on both tasks.

Comment 3. Multiple intermediate entities and answer uniqueness

Section 4.2 uses a single intermediate entity, e₁, for notational simplicity. In the actual inference procedure, we retain all candidate entities predicted at the first hop. For each candidate eⱼ, we instantiate the [entity] placeholder in the second subquestion to obtain a separate query q₂(eⱼ), then retrieve evidence and generate an answer independently. For example, candidates eₐ and eᵦ produce queries q₂(eₐ) and q₂(eᵦ).

In the current evaluation, we score each question across all candidate answers without requiring the system to select a single final prediction. We count a question as correct if at least one branch produces an answer that exactly matches the gold answer. If none of the branches produces an exact match, we count the question as incorrect.

In a user-facing setting, if different branches produce different answers, the system can present those answers to the user, who can select the one that best fits their intent and context. The benchmark evaluation omits this user confirmation step and checks only whether at least one candidate answer exactly matches the gold answer.

Thanks for the response. I have decided to maintain my original score.

Reviewer_ziJWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KMMG-Agent, a framework for multi-hop question answering over personal memories and external knowledge. The method represents memory events as timestamped graph edges and combines temporal filtering with entity-guided retrieval across two hops. The paper also constructs a multi-hop question-answering benchmark that integrates memory and external knowledge and covers four distinct reasoning paths. Experiments show improvements over the reported baselines on the memory-related tasks.

Critical strengths
  1. The paper addresses an underexplored problem of multi-hop reasoning across personal episodic memories and external Wikipedia knowledge.
  2. The Mem-WikiQA benchmark is a useful contribution, covering four types of cross-source reasoning with human-filtered samples.
Critical weaknesses
  1. Dataset construction heavily relies on LLMs for triple extraction and question generation. Although manual filtering is performed, latent LLM hallucinations may still contaminate the dataset.
  2. The pipeline involves several LLM-based steps, such as question decomposition, triple extraction, and entity alignment. Errors in these steps may affect the final results, but the paper does not provide a detailed analysis of error propagation.
  3. The evaluation relies only on Exact Match (EM), making it difficult to determine whether errors come from retrieval or reasoning.
Questions for authors
  1. Mem‑WikiQA heavily uses LLMs for triple extraction and question‑answer pair generation. Could you report the approximate error rate of LLM outputs before manual filtering? How do you mitigate potential hallucinations from LLMs in dataset construction?
  2. Many intermediate steps rely on LLMs. Could you provide error analysis to show where the model fails dominantly? How does error propagation across pipeline stages affect overall performance?
  3. Could you add retrieval‑level metrics (such as fact recall) to separate failures caused by retrieval versus LLM reasoning? Pure EM cannot fully diagnose model behaviours.
  4. How does the proposed memory multi‑graph scale with a large growing volume of memory events? What are the main computational or storage bottlenecks for real‑world deployment?
2 comments

Response to Reviewer

We sincerely thank the reviewer for the constructive questions and valuable suggestions. We address each concern below and provide additional experimental evidence and clarifications.

Comment 1. Dataset Quality and Hallucination Mitigation

To reduce the risk of LLM hallucinations during dataset construction, we ground QA generation in the original evidence and manually review the resulting question–answer pairs. We first extract triples from EgoLife video captions and Wikipedia passages. We retain the source text and, for memory events, the corresponding timestamps so that the extracted facts can be traced back to their sources. We then generate candidate QA pairs from explicit two-hop reasoning paths, incorporating temporal information whenever a path involves a memory event. This grounds generation in the selected facts and reduces reliance on the LLM’s internal knowledge. For each of the four task types, we manually reviewed all 1,600 candidate QA pairs against the original captions and Wikipedia passages. We checked the factual accuracy of the supporting evidence, the consistency of the intermediate entity across the two hops, the validity of the reasoning chain, and the accuracy of any temporal constraints. We excluded candidates with identified errors and retained 500 QA pairs per task type, yielding 2,000 examples in total. By extracting triples from the original sources, generating candidate QA pairs from explicit two-hop reasoning paths, and manually checking each pair against the source evidence, we reduce the risk that LLM hallucinations and other upstream errors propagate into the final dataset.

Comment 2. Error analysis and propagation across stages

We manually analyzed 50 failed cases from each of the four task types, for a total of 200 cases. Among the failures examined, errors in retrieving memory or knowledge facts accounted for 68%, answer generation errors accounted for 24%, and errors in question decomposition or temporal parsing accounted for the remaining 8%. To examine how errors propagate, we replaced the predicted first-hop entity in each case with the manually verified correct entity and reran the subsequent stages. Of the 50 failed cases per task type, this intervention yielded a correct final answer in 29 Memory-to-Memory cases, 30 Memory-to-Wiki cases, 25 Wiki-to-Memory cases, and 23 Wiki-to-Wiki cases. These results show that providing the correct intermediate entity can recover a substantial fraction of the failures examined. They also illustrate how a first-hop entity error can affect the formulation of the second-hop query, the evidence retrieved, and ultimately the final answer.

Comment 3. Retrieval-level evaluation

To complement end-to-end exact match (EM), we evaluated the retrieval of supporting facts on 100 questions from each of the four task types. We computed first-hop Fact Recall over all 100 questions per task type. For the second hop, we computed Fact Recall only on questions for which the model correctly predicted the first-hop intermediate entity in its original run. This conditional metric measures how often the second retrieval step finds the required evidence when the model has correctly identified the intermediate entity. These metrics complement end-to-end EM by reflecting the evidence recall performance across different tasks at each hop in retrieval, thereby providing a basis for identifying retrieval bottlenecks.

Table 1. Fact Recall across the four task types.

Task TypeFirst-Hop Fact RecallSecond-Hop Fact Recall
Memory-to-Memory82%68%
Memory-to-Wiki84%86%
Wiki-to-Memory68%74%
Wiki-to-Wiki70%72%

Comment 4: Scalability and deployment bottlenecks

The memory multigraph can grow by appending event records: timestamped triples are extracted from incoming data, added to the graph in the same format as existing records, and encoded as vectors. The main potential bottlenecks are the computational cost of extracting and encoding new triples, the cost of searching larger candidate sets when a query spans a broad time window, and the latency of sequential LLM calls. At the data scale and under the evaluation settings used in our runtime experiment, our method answered the evaluation questions in less time than the baselines included in the comparison. We also analyzed the execution time of each component; the detailed results are presented in Table 2. For large-scale deployment, optimizations such as temporal indexing, incremental index updates, and approximate vector search could be adopted. These remain directions for future engineering optimization, and their effectiveness has not yet been validated in this work.

Pipeline StageTime Cost Percentage
Sub-question Decomposition22.21%
First-Hop Retrieval4.37%
Entity Generation4.20%
Second-Hop Retrieval47.28%
Answer Generation21.94%

I appreciate the authors' additional analysis and clarifications. However, some concerns regarding error propagation and the reliability of the multi-stage pipeline remain. These issues still limit my assessment of the method, so I will maintain my original rating of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.