- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work tackles the challenge of multi-hop question answering requiring integration of personal temporal memory data from first-person video captions and static external knowledge from Wikipedia. The authors propose KMMG-Agent, a framework that represents memories as a multi-graph with timestamped edges to preserve repeated events at different times, combining query-specific temporal filtering with intermediate entity-based chained retrieval. They also introduce Mem-WikiQA, a new benchmark with 2,000 two-hop questions covering four evidence-source paths (Memory-to-Memory, Memory-to-Wiki, Wiki-to-Memory, Wiki-to-Wiki). Experimental results show KMMG-Agent outperforms existing baselines by 11.8%-33.2% in Exact Match accuracy on memory-related tasks.
Critical strengths
Critical Strengths
1.The timestamped edge representation for temporal memories is an interesting idea that could potentially be applied in other memory modeling contexts.
2.The Mem-WikiQA benchmark addresses a relevant problem space of cross-source reasoning.
Critical weaknesses
Critical Weaknesses
1.Severe limited reasoning scope: The benchmark is artificially restricted to only two-hop reasoning, which is far less complex than real-world multi-hop QA scenarios. This severely limits the practical impact and generalizability of the work.
2.Unaddressed entity alignment critical flaw: The paper completely ignores entity alignment errors between memory and Wikipedia sources, which is a fundamental challenge in cross-source reasoning systems. Without addressing this, the proposed approach cannot be deployed in real-world applications.
3.Insufficient experimental validation: The experiments only compare with a limited set of baselines and do not include ablation studies that properly isolate the contribution of each component. The reported improvements are not convincingly demonstrated to be due to the proposed multi-graph design rather than other confounding factors.
4.Lack of novelty: The core ideas of multi-graph memory representation and temporal filtering have been explored in prior work, with only incremental modifications presented in this paper.
Questions for authors
None, as the paper has fundamental limitations that cannot be addressed with minor revisions.
Response to Reviewer
We thank the reviewer for the constructive comments.
Comment 1. Limited reasoning depth of the benchmark
We thank the reviewer for this important comment. We acknowledge that real-world multi-hop QA often involves more complex reasoning processes than two-hop scenarios, requiring the integration of information across multiple entities, relations, and sources.
Our focus on two-hop reasoning is an intentional choice to validate the effectiveness of our multi-graph memory representation and temporal filtering mechanism in a controlled setting. This setting provides a manageable yet representative scenario for evaluating cross-source information integration, relational reasoning, and temporal-aware retrieval, while allowing us to better isolate the contributions of our proposed mechanisms.
We emphasize that two-hop reasoning reflects the scope of our current evaluation rather than an inherent limitation of the framework. Extending the evaluation to more complex scenarios involving three or more hops, as well as longer reasoning chains and more diverse temporal dependencies, is an important direction for future work.
Comment 2. Entity Alignment Between Memory and Wikipedia Sources
We appreciate the reviewer's valuable comment regarding entity alignment between memory and Wikipedia sources. We agree that entity matching across heterogeneous information sources is an important challenge in cross-source reasoning systems. In our approach, retrieval is performed through two complementary components: semantic retrieval and entity retrieval. Specifically, semantic retrieval identifies relevant information based on semantic similarity, enabling the system to retrieve contextually related content even when different sources use different expressions. Meanwhile, entity retrieval focuses on identifying and retrieving relevant entities, helping establish entity-level correspondences across different information sources. By combining these two retrieval mechanisms, our approach considers both semantic relevance and entity-level information, rather than relying solely on exact entity name matching.
Comment 3. Additional Ablation Studies
1. Ablation on Retrieval Mechanisms.
First, we conducted experiments with three retrieval configurations: Entity-only Retrieval, Semantic-only Retrieval, and Hybrid Retrieval (combining entity-based and semantic retrieval). These experiments allow us to examine the individual contributions of entity-based and semantic-based retrieval, as well as the effectiveness of combining these complementary mechanisms.
Table 1. Exact-match accuracy (%) for second-hop retrieval strategies under a matched evidence budget.
2. Ablation on Evidence Sources.
Second, to further investigate the contribution of evidence from different sources, we conducted additional experiments by removing evidence from individual sources (i.e., personal memory or Wikipedia) and removing evidence from both sources. These experiments allow us to assess the importance of each evidence source and evaluate the impact of integrating cross-source evidence on overall performance.
Table 2. Exact-match accuracy (%) under different evidence availability conditions.
Overall, these additional experiments provide a more systematic evaluation of the proposed retrieval mechanisms and evidence integration strategy, offering further empirical support for the design choices underlying our framework.
Comment 4. Methodological Contributions and Novelty
We sincerely thank the reviewer for the valuable comments regarding the novelty of our work. Our contribution lies not in proposing individual techniques in isolation, but in integrating them into a unified framework for cross-source reasoning.
Specifically, our framework focuses on connecting heterogeneous information sources, particularly memory and Wikipedia, through three complementary mechanisms: (1) Multi-graph memory representation to organize and connect information from different sources; (2) Entity-based and semantic-based retrieval to identify relevant information by considering both entity associations and semantic relevance; and (3) Temporal filtering to select relevant evidence based on temporal information.
Together, these mechanisms provide an integrated approach to organizing, retrieving, and utilizing heterogeneous evidence for cross-source reasoning.