acceptodds
Under review as a conference paper at ICLR 2027

DG-Mem: Dual-Graph Memory Retrieval for Multi-User Agents

Abstract

Long-term memory enables LLM agents to build on past interactions, but most existing systems are designed around a single user. Shared assistants in enterprise channels, project discussions, and organizational workflows must instead serve multiple users through a common conversational history. This introduces three challenges: distributed and heterogeneous evidence, coexisting and evolving user states, and asker-dependent relevance. Content-oriented retrieval and memory consolidation can miss cross-user connections or obscure the attribution needed to distinguish valid information simultaneously. We propose , a dual-graph framework that couples shared content organization with user-conditioned evidence selection. A content-centric graph connects evidence through semantic, conversational, and temporal relations, while a user-context graph preserves speaker-grounded distinctions through authorship and interaction structure. Aligned message anchors connect these complementary views. Asker-conditioned cross-graph retrieval accesses relevant evidence across participants, and query-conditioned refinement selects supporting messages according to user context and the question's specific requirements. The framework also supports conventional single-user memory, providing a common foundation for personal and shared histories. Memory construction and retrieval do not require generative-LLM calls and incur zero generative-LLM token cost before answer generation. Experiments on GroupMemBench and LoCoMo demonstrate state-of-the-art overall performance in multi-user and conventional single-user settings, respectively. Our code is available at https://anonymous.4open.science/r/DG-Mem.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-User Agents, agent memory, dual graph

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_DXndOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of long-term memory retrieval in multi-user shared conversation histories, with a focus on evidence selection errors where the retrieved content is relevant but is associated with the wrong user or context. The authors propose DG-Mem, which aligns two complementary views through shared message nodes: a content graph organizes entity associations, replies, and temporal relations, while a user-context graph preserves authorship, roles, and discussion scope. Retrieval first discovers candidate evidence in the content graph, then propagates information over the user-context graph using the asker's information, and finally performs query-conditioned reranking to select the messages provided to the answering model.

Critical strengths
  1. In multi-user conversations, different users may hold information about the same object that is simultaneously valid but applicable to different scopes. The paper clearly distinguishes content relevance'' from asker relevance'', rather than treating multi-user memory simply as a larger-scale text retrieval problem. This problem formulation is practically meaningful and provides a clear motivation for the proposed method.

  2. The two graphs establish associations over the same set of messages, allowing the system to retrieve information provided by other participants while preserving authorship, reply relationships, and discussion context. Compared with compressing statements from multiple users into a single global conclusion, this design preserves the evidence needed to distinguish different users' states and also makes the basis of an answer easier to inspect. The shared message nodes enable direct and traceable evidence transfer between the two views.

  3. The method uses entity extraction, local encoders, and graph propagation for memory construction and retrieval, without repeatedly invoking a generative LLM to extract, summarize, or rewrite memories. Under the experimental setting reported in the paper, memory construction takes \textbf{0.76 hours} and requires \textbf{345 MB} of storage. This design substantially reduces construction overhead while maintaining strong answer quality, making it attractive for shared assistants that continuously accumulate large volumes of messages.

Critical weaknesses
  1. The User-Context Graph organizes messages using \texttt{channel}, \texttt{phase}, \texttt{topic}, \texttt{role}, and reply relationships. While authorship and time are typically directly observable, the sources of topic, phase, and role require more explicit clarification. These fields already provide part of the discussion-scope information, whereas recovering the correct discussion context is precisely one of the problems addressed by the paper. It is therefore unclear how much of the system's advantage remains when such structured information is unavailable. In particular, when a message has no explicit reply target but refers back to previous content using expressions such as that project'' or the previous plan,'' the paper does not clearly explain how the associations between messages are recovered.

  2. The content graph and user-context graph can connect relevant messages, but shared entities, authors, or discussion scopes alone cannot distinguish whether a message proposes an idea, rejects a proposal, or confirms a final decision. For example, I suggest postponing it until Friday'' and The postponement was not approved; we will proceed on Wednesday'' may have very similar graph neighborhoods. Temporal edges that decay with the time interval between messages also do not directly indicate that a fact has been superseded. Although the paper achieves strong results on information-update tasks, it does not sufficiently explain which component performs this state distinction and under what circumstances outdated or unconfirmed messages may still be ranked ahead of the correct evidence.

  3. The system uses lexical rules to identify self-reference, temporal and state cues, as well as the fields requested by a query, and uses these signals to adjust propagation and reranking weights. Field matching may also restrict the evidence entering sentence-level reranking, meaning that these rules can substantially affect which information is ultimately selected. The paper provides some weights and hyperparameters, but does not sufficiently explain how the rules are designed, how indirect expressions are handled, or whether there is a reliable fallback mechanism when detection fails. The concern is not the use of rules itself, but rather that it is currently difficult to assess how strongly the results depend on particular question formulations and field expressions.

Questions for authors
  1. Please clarify the sources of \texttt{phase}, \texttt{topic}, and \texttt{role}, respectively. Are they fields directly available in the original records, information provided by the data-generation process, or attributes inferred by an additional procedure? When these fields are missing, how are the scope nodes constructed, and how does system performance change?

  2. For interleaved discussions within the same channel, suppose later messages refer to previous content only through expressions such as that plan'' or let's go with what they said,'' without a \texttt{reply_target}. What mechanism does the system use to associate these messages with the correct historical messages? Are entity aliases and cross-message coreference explicitly handled?

  3. In cases involving multiple modifications, rejections, and reconfirmations within a short period, which component is responsible for determining the validity status of each message? Could the authors provide the candidate rankings at different stages for such cases, illustrating how the system retrieves the final decision and its applicable scope rather than simply passing conflicting evidence to the answering model?

  4. Since LoCoMo lacks enterprise-specific roles, channels, and discussion phases, which nodes and edges are retained in (G^U), and how are the scope and asker defined? Does this experiment use the full cross-graph pipeline, or does it primarily rely on the content graph?

5 comments
AuthorsOpenReview ↗

Response to Reviewer DXnd (1/2)

We sincerely appreciate your careful review and positive assessment of our problem formulation, traceable evidence transfer, and efficient memory construction, as well as your valuable insights. Below are detailed responses to each concern.

Q1: Performance without role, phase, and topic metadata.

In GroupMemBench, author, role, channel, phase, and topic are supplied with the synthetic conversation records. DG-Mem uses these fields directly and does not infer them or make generative-LLM calls to create them. These fields are also included in the answer-generation inputs of the official GroupMemBench baselines and the other compared methods. To test our dependence on role, phase, and topic metadata, we removed these three fields and re-evaluated DG-Mem on GroupMemBench.

MethodAverageMulti-HopUpdateAmbiguityImplicitTemporalAbstention
DG-MEM55.3052.2043.0032.0857.1453.0988.49
No phase/topic/role52.7548.9039.2521.7057.1452.4790.65
Hindsight46.0142.3117.7637.7440.8254.9477.06
SimpleMem28.7225.828.4112.2620.4127.7864.75
MemGPT28.1522.5317.7620.7528.5712.4277.98

Removing role, phase, and topic reduces overall accuracy from 55.30% to 52.75%, retaining 95.4% of the full model's accuracy. This variant still exceeds Hindsight, the strongest baseline, by 6.74 percentage points. DG-Mem therefore retains a substantial overall advantage without these three types of metadata. We have added this comparison and the accompanying discussion to Section 4.6 of the revised version.

Q2: Indirect references without an explicit reply target.

DG-Mem uses reply links when available, but does not require them to retrieve a message. It does not perform explicit cross-message coreference or general alias resolution. Instead, it gathers candidates through entity, dense, and lexical signals in the content graph, temporal connections, context-aware BM25, and participant/scope propagation in the User-Context Graph, followed by sentence-level reranking. These signals can help retrieve messages containing indirect references when their content or interaction context provides additional identifying cues.

The following Technology example illustrates retrieval of an update without an explicit reply link:

Case trace
QuestionWhat is the current plan for handling transition metadata?
Gold evidenceMsg_3731 : “Change to the earlier decision: transition metadata is no longer required now ... keep the current payload contract, map transport failures to validation-only handling ... then do the schema update after.”
DG-MEM retrievalMsg_3731 at rank 4 (user_context = 0.529, context_match = 1.00, final score 0.701).
PredictionKeep the current payload contract, use validation-only handling for transport failures, and defer transition metadata to a later schema update. Correct.

The update was posted independently by another participant and refers to an “earlier decision” without an explicit reply link. DG-Mem nevertheless retrieves the update and answers correctly. This illustrates how informative message content and interaction context can support retrieval without reply links. A generic reference such as “that plan,” with no distinguishing content or interaction cues, may still remain ambiguous.

AuthorsOpenReview ↗

Response to Reviewer DXnd (2/2)

Q3: Retrieving the final decision from conflicting updates.

Evidence-status cues are used in query-conditioned candidate refinement, together with content relevance, context-aware BM25, user-context propagation, asker/scope compatibility, field matching, and sentence-level reranking. Under the default query-aware configuration, the evidence-status weight is set to zero when no temporal, state, or exact-answer query cue is detected. Field matching is disabled when no specific answer field is identified, and the remaining weights are normalized. For a state-sensitive query, a candidate without a recognized status cue receives a neutral status score.

On GroupMemBench's Update category, DG-Mem achieves 43.00% accuracy, exceeding the strongest baseline by 15.90 percentage points. To examine how conflicting updates are ranked, we traced Technology question knowledge_update_3, whose history contains an initial scope, an earlier hold decision, a counterproposal, and a revised final decision.

Case trace
QuestionWhat is the current plan for convention tagging in QA?
Ground truthUse convention tagging in the primary checklist only for the flagged rule family; keep broader channel-wide tagging deferred.
Conflicting historyMsg_1558: “hold the convention tags” (obsolete).
Msg_2328: “tag convention as a secondary dimension” (superseded).
Msg_4055: “move convention tagging into the primary QA checklist for that flagged rule family; keep broader channel-wide tagging deferred” (gold evidence; no reply target).
CandidateContent GraphCross-graph baseFinal
Msg_856: initial scope244
Msg_1558: hold convention tags123
Msg_2328: use convention as a secondary tag332
Msg_4055: final scoped decision; no reply411

The content graph ranks the earlier hold decision first and the final scoped decision fourth. Cross-graph fusion moves the final decision to first place before the reader is called. Refinement keeps it first and lowers the hold decision from second to third. The reader then returns the correct answer with its applicable scope. This case shows that retrieval prioritizes the final decision before answer generation, while retaining earlier statements as context.

Q4: LoCoMo setting.

LoCoMo does not provide enterprise-specific roles, channels, or workflow phases. We use session IDs for the channel and phase fields, set the topic to “LoCoMo conversation” and the role to “conversation participant,” and leave asking_user_id empty.

Both graph retrieval stages remain active. Content-graph candidates seed user-context propagation through shared message anchors, and the final candidate pool combines content, context-aware BM25, and user-context PPR results before sentence-level reranking. The LoCoMo configuration emphasizes content relevance and semantic reranking, while the User-Context Graph retains session and participant structure. Asker-specific seeding is inactive because no asker identity is supplied. We have clarified these settings in the revised version.


Thanks again fo your time, efforts, and constructive feedback. We have added your valuable feedback to the revised version. If you have any additional questions or suggestions, we would be happy to discuss them.

Thank you again for your time and thoughtful review. With only three days remaining in the discussion period, we would greatly appreciate your feedback on whether our rebuttal has addressed your concerns. If any concerns remain, we would be happy to provide further clarification.

Reviewer_2CFkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DG-MEM, a dual graph memory framework for conversations with multiple users. It aligns a content graph with a user context graph, retrieves evidence conditioned on the asker, and refines candidates conditioned on the query, without generative LLM calls. It reports strong accuracy on GroupMemBench and LoCoMo with large reductions in construction time and storage.

Critical strengths

Task accuracy and realism: Shared collaborative environments are indeed a problem worth studying, and they are also indeed a blind spot in current memory systems. Research on this aspect is still relatively scarce.

Reasonable and clear framework design: The dual-graph structure design concept tailored to the scenario is reasonable and strongly relevant to the task.

Engineering efficiency: The graph construction process does not use LLMs, which offers advantages in terms of time and storage.

Comprehensive and reasonable experiments: The experiments are comprehensive and reasonable, and the results are convincing.

Critical weaknesses

1.Some hyperparameters lack sensitivity analysis—namely, ω_M, the receives weight, and the coefficient of 0.60 used for query-matched role and scope nodes, with at most eight scope nodes activated. These parameters may also affect the results and therefore require a certain degree of analysis.

A2.lthough no LLM is used in the construction, introducing NER and embedding models still incurs some overhead, which should be reported transparently.

Questions for authors

see above

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2CFk

We sincerely thank you for your positive assessment of our work’s practical relevance, dual-graph design, construction efficiency, and comprehensive experiments, as well as your constructive suggestions. We address your comments in detail below.

W1: Hyperparameter Sensitivity Analysis

Thank you for pointing out these parameters. We ran additional sensitivity experiments on the full GroupMemBench evaluation set, varying one hyperparameter at a time while keeping the other settings unchanged from the main experiments.

The entries of ωM\boldsymbol{\omega}^{M} are the fusion weights for content-graph PPR, dense retrieval, and BM25, respectively. The role/scope seed coefficient controls the raw personalization mass contributed by query-matched role and scope nodes before joint normalization. The scope-node limit specifies the maximum number of query-matched scope nodes included in the seed vector.

Table R1. Hyperparameter sensitivity on GroupMemBench. Bold entries mark the default settings and their results. Δ denotes the accuracy change from the default configuration in percentage points.

HyperparameterSettingAccuracy (%)Δ (pp)
ωM\boldsymbol{\omega}^{M}(0.40, 0.30, 0.30)52.21-3.09
(0.50, 0.25, 0.25)55.30—
(0.60, 0.20, 0.20)50.87-4.43
Role/scope seed coefficient0.4052.62-2.68
0.6055.30—
0.8053.15-2.15
Maximum activated scope nodes452.75-2.55
855.30—
1652.21-3.09

The default settings achieve the highest accuracy within each tested range. For content-score fusion, increasing the PPR weight reduces the relative contribution of dense and lexical matching, while decreasing it places less emphasis on graph-propagated relevance. Neither change improves upon the default combination of (0.50, 0.25, 0.25). Changes to the role/scope seed coefficient and scope-node limit also yield lower accuracy than their defaults.

Although accuracy varies across configurations, every tested setting remains above 50%. The lowest result, 50.87%, still exceeds Hindsight, the strongest baseline in Table 1, by 4.86 percentage points. Thus, DG-Mem retains its overall advantage across the tested parameter variations. We will include these additional results and analysis in the appendix.

W2: NER and embedding overhead

DG-Mem uses local spaCy NER and the local all-MiniLM-L6-v2 encoder during memory construction. We measured their wall-clock time using the first 5K, 15K, and 30K messages from the Finance history. NER ran on the CPU, while embedding ran on an NVIDIA A10 GPU with a batch size of 64. These runs used no generative-LLM calls or tokens.

MessagesTotal construction (s)NER (s)Embedding (s)
5,00035.9925.502.96
15,00096.2377.738.70
30,000180.94151.1717.02

At 30K messages, NER took 151.17 seconds and embedding took 17.02 seconds, accounting for 83.5% and 9.4% of total construction time, respectively, or 93.0% combined. Their amortized costs were approximately 5.04 ms and 0.57 ms per message. NER was the main construction cost, and the complete construction run finished in 180.94 seconds, approximately three minutes.


Thanks again for your time and constructive feedback. We have incorporated the above analysis and experiments into the revised manuscript. Please feel free to reach out if you have further questions or suggestions.

Thank you again for your time and thoughtful review. With only three days remaining in the discussion period, we would greatly appreciate your feedback on whether our rebuttal has addressed your concerns. If any concerns remain, we would be happy to provide further clarification.

Thank you for your response, and I have no further questions.

Thank you for your thoughtful feedback and for acknowledging the additional experiments. We greatly appreciate your support and these results have already been incorporated into the revised manuscript. Your encouragement means a lot to us as we refine this work.

Best wishes,

All authors

Reviewer_mg8HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DG-Mem studies memory retrieval in multi-user conversations. It links a content graph with a user-context graph, using the question and who is asking to retrieve and rerank messages. No generative LLM calls during memory building or retrieval.

Critical strengths
  1. Useful problem, sensible design.

  2. Promising results and practical cost savings.

Critical weaknesses
  1. Table 1 may mix different evaluation sets? DG-Mem’s loader shows no corresponding filtering step. Please confirm that all methods were evaluated on the same question IDs.

  2. LoCoMo changes more than the retriever. it can change F1/BLEU even with identical evidence. Retrieved units also span up to six turns. §4.1/A.5 don’t confirm that baselines used the same prompts, category info, or evidence-token budget. Same LLM + Top-5 isn’t enough.

  3. “64 candidates” is a branch limit, not the actual reranking budget. A.5 says 64 in both stages, but the code merges 32 dense + 64 PPR + 64 BM25 candidates, then up to 64 + 64 + 64 in the second stage. The merged pools can therefore reach 160 and 192 messages, depending on overlap. This changes both candidate coverage and scoring cost, so Table 4 should report actual merged counts alongside the branch limits.

Questions for authors
  1. Were the question sets identical?

  2. Were the reader settings matched?

  3. What does each ablation actually switch off?

6 comments
AuthorsOpenReview ↗

Response to Reviewer mg8H (1/2)

Thank you for acknowledging our work’s critical strengths of a useful problem and sensible design, along with promising results and practical cost savings, and for identifying these evaluation and reporting issues, we provide detailed responses to your comments as follows.

W1 & Q1: Question-set consistency in Table 1

We used all questions in the released GroupMemBench dataset, covering four domains and six question types, and followed the official evaluation protocol. We did not apply any additional filtering or modify the questions. (Some baseline results are taken from the official GroupMemBench paper.)

The solvability filtering described in the benchmark documentation for Finance and Technology was performed during dataset construction and is already reflected in the released question files. Our loader reads these files directly, as do the official baseline loaders, so no additional filtering is needed at evaluation time. We compared the loader outputs and confirmed that the question IDs, question text, and reference answers match exactly. We will clarify this distinction in the appendix.

W2 & Q2: LoCoMo reader settings.

It is important to ensure consistent reader settings, as using the same LLM and top-k alone does not guarantee an identical reader setup. The baseline scores in Table 2 were taken from the results reported by GAM. We ran an additional GPT-4o-mini evaluation using the prompts from GAM's released LoCoMo code and the same evidence-token budget. We updated the experiment configuration without changing the DG-Mem implementation. The last row reports this additional run; the remaining rows reproduce the results in Table 2.

MethodAverage F1Average BLEU-1Single Hop F1Single Hop BLEU-1Multi Hop F1Multi Hop BLEU-1Temporal F1Temporal BLEU-1Open Domain F1Open Domain BLEU-1
LONG-LLM37.3129.5746.6837.5429.2322.7625.9719.4216.8713.70
RAG44.7339.4052.4547.9427.5020.1346.0740.3523.2317.94
HippoRAG47.9241.0254.8448.8433.5925.4648.1739.3228.5923.89
A-Mem39.6532.3144.6537.0627.0220.0945.8536.6712.1412.00
Mem045.1035.9047.6538.7238.7227.1348.9340.5128.6421.58
MemoryOS42.8435.5448.6242.9935.2725.2241.1530.7620.0216.52
LightMem38.4434.3641.7937.8329.7824.8043.7139.7216.8913.92
SimpleMem48.0241.6853.4847.5936.9328.8051.3045.2921.7815.67
CompassMem52.1844.0957.3649.7938.8427.9857.9650.5126.6120.01
GAM53.7547.5157.7552.1042.2934.4459.4553.1133.3026.97
DG-Mem (original)60.7953.7569.5563.3244.2433.7960.6154.1133.1527.39
DG-Mem (GAM prompts/budget)57.7751.1967.9961.9640.3032.0254.1847.4031.5625.81

With GAM's prompts and evidence-token budget, DG-Mem achieves an average F1 of 57.77 and BLEU-1 of 51.19, exceeding GAM by 4.02 and 3.68 points, respectively. Its overall advantage therefore remains under these matched conditions.

W3: Actual candidate and reranking budgets

Thank you for pointing out the ambiguity in our description of candidate_pool = 64. It specifies a per-branch limit, not a hard cap on the total number of unique messages entering refinement.

Under the original default configuration, the first stage merges up to 32 dense-retrieval candidates, 64 content-PPR candidates, and 64 content-BM25 candidates. Their union can contain up to 160 unique messages, which are ranked to retain 64 content proposals. The second stage merges these proposals with up to 64 context-aware BM25 candidates and 64 user-context PPR candidates, giving an upper bound of 192 unique messages before refinement. The actual counts depend on overlap between the candidate sets.

We measured these counts over all GroupMemBench questions across the four domains and six question types, using the full histories.

QuantityMean per queryMaximum
Unique merged first-stage messages117.97128
Content proposals passed to the second stage64.0064
Unique messages entering second-stage refinement118.31142
Final selected evidence messages10.0010

We will clarify the definition of candidate_pool in the appendix and include these measured counts, distinguishing per-branch limits, actual merged candidate counts, and the final evidence budget.

AuthorsOpenReview ↗

Response to Reviewer mg8H (2/2)

Q3: Ablation studies

w/o Content-Centric Memory Graph. This variant completely disables the Content-Centric Memory Graph. Specifically, we remove message–entity connections, reply edges, temporal edges, and same-phase edges; we do not run personalized PageRank over the content graph; and we disable the content-graph fusion of dense retrieval, content BM25, and graph propagation. The User-Context Graph, context-aware BM25, asker and context seeds, user-graph propagation, and query-conditioned refinement remain active.

w/o User-Context Graph. This variant completely disables the User-Context Graph and all associated retrieval paths. We remove the connections between messages and their authors, mentioned users, conversational scopes, and reply parents, as well as the user–role and user–scope connections. Context-aware BM25, User-Context Graph PageRank, asker seeds, explicit-user seeds, and role/scope seeds are also disabled. The final evidence depend on the Content-Centric Memory Graph.

w/o Asker-Conditioned Cross-Graph Retrieval. This variant retains both graphs but disables the asker- and query-conditioned retrieval process that connects them.

w/o Query-Conditioned Candidate Refinement. This variant preserves the complete dual-graph candidate-generation process and asker-conditioned cross-graph propagation, but disables all question-specific fine-grained refinement.


Once again, we sincerely appreciate your time, efforts, and constructive feedback. We have added your valuable feedback to the revised PDF. If you have any additional questions or suggestions, we would be happy to discuss them.

Thank you again for your time and thoughtful review. With only three days remaining in the discussion period, we would greatly appreciate your feedback on whether our rebuttal has addressed your concerns. If any concerns remain, we would be happy to provide further clarification.

Reviewer_sQhJOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the following problem: in group-chat settings, an LLM agent / assistant with full access to an organization’s information provides personalized question answering to the users within that organization, and it needs a system that helps maintain its memory. The paper’s contribution is to further elaborate on two of the problems already raised by the recent GroupMemBench benchmark: group structure being flattened, and question answering not being conditioned on the speaker / asker. The paper frames these as content-oriented retrieval and memory consolidation that “miss cross-user connections or obscure the attribution needed to distinguish simultaneously valid information,” and digs into them further. To address them, it proposes a solution that combines a content-centric graph with a user-context graph. Using the result of one PPR as the seed for another PPR is logically simple and elegant, and it is an intuitively effective way to retrieve over two graphs jointly, bringing a cross-layer PageRank idea already known in network science (Multiplex PageRank) into multi-user memory retrieval. This is also one of the earlier academic attempts to handle this important group-chat scenario. Apart from possibly being limited by the scope of the dataset itself, which does not consider permission management within organizations, the work has potential value for Teams/Feishu-style products and for organizations’ self-built AI question-answering systems. It is also training-free, and even LLM-free (rarely seen in the field's literature) during memory construction. The paper compares against existing memory systems designed for single-user scenarios and reports leading results, about 9 points above the strongest baseline on GroupMemBench, with most of the gain on knowledge-update questions and most of it attributed to the user-context graph in the ablation; it also remains competitive on the dyadic LoCoMo benchmark. Part of this margin may come from DG-Mem using author, role, and scope metadata in retrieval, which the baselines only see at answer time.

Critical strengths
  1. The problem setting is important and well-motivated. A shared assistant serving many users from one common history is a realistic need, and the paper clearly identifies what breaks when single-user memory is applied to it: evidence that is only connected across users, and different users’ states that are valid at the same time.

  2. The method is simple, clean and intuitive. The content graph finds what is related, the user-context graph finds whose it is, and seeding the second PPR with the results of the first connects the two in a natural way. Each component has a clear role, and the ablation supports the core claim: removing the user-context graph causes the largest drop (15.0 points).

  3. It is easy to deploy. Memory construction and retrieval need no training and no generative-LLM calls, the retrieval is deterministic, and the memory layer is independent of the answering model. In an organizational setting, this also means the chat history does not have to be sent message by message to an LLM during memory construction, which potentially reduces cost.

Critical weaknesses
  1. The method relies too heavily on information available in the synthetic data. In my personal understanding, GroupMemBench expects researchers to use author and role information, but using phase and topic labels may not match its original design intent. GroupMemBench’s official baselines index only message content during retrieval, while DG-Mem uses author, role, channel, phase, and topic during both memory construction and retrieval, so the comparison is not entirely fair. If this information were also available in practice, that would be a good thing, since the method would be making fuller use of the available constraints. However, it is hard to imagine a group-chat setting that naturally provides phase and topic as ground-truth labels. Such information may exist in a few settings such as ticket-based collaboration tools; for example, in Jira, comments naturally belong to an issue, an epic, and a sprint. But that is not the free-form group-chat setting the paper targets. If an LLM were used to annotate these labels instead, the label quality would be questionable, and it would also contradict the authors’ own claim of zero LLM calls during memory construction. I suggest that the authors keep only the channel in the scope triple (since groups/channels should exist in most real settings), drop the phase annotations from graph construction, and rerun the experiments. This should be feasible: the method already handles missing fields, since phase edges are built only when the phase is non-empty, and the method has been run on LoCoMo, which has no such labels. Alternatively, the authors could replace the ground-truth labels with LLM-generated annotations and report the results. This would not affect the authors’ core dual-graph contribution, and it would make the method more broadly applicable.

  2. There is no efficiency analysis of the retrieval and question-answering stage, i.e., the whole process from the moment a user query arrives to the moment the final answer is sent. A memory system can be roughly divided into online, sequential memory construction and query-time retrieval and question answering, where each question is handled independently. The paper’s efficiency analysis only covers token consumption and wall-clock time in the memory construction stage. Consider that: (1) retrieval and question answering are what a memory system ultimately serves; (2) some graph-based or agentic methods, such as GraphRAG, GAM, and A-MEM, are known to have high query-time latency or token cost; and (3) DG-Mem itself runs two PPR passes over a graph with tens of thousands of message nodes, followed by cross-encoder reranking, for every query. Taking these three points together, I believe this comparison is something the community will pay close attention to.

  3. The way results are reported in Tables 1 and 2 (and likewise in Tables 5–12 in the appendix). Both tables compute differences relative to a relatively weak method, text-embedding-3-large and LONG-LLM respectively. I suggest following the common practice of highlighting the best-performing baseline in each column and reporting the proposed method’s change relative to it, even when the proposed method is not higher. For example, on GroupMemBench, DG-Mem is not the best on Temporal or Term Ambiguity, but this is not easy to see in the current format.

Questions for authors

I have no further questions beyond the three weaknesses above.

8 comments
AuthorsOpenReview ↗

Response to Reviewer sQhJ

We sincerely thank you for recognizing the practical relevance of our problem setting, the simplicity and intuitiveness of our dual-graph design, and its ease of deployment, as well as for your constructive suggestions. We address your comments in detail below.

W1. Dependence on phase, topic, and role metadata.

We thank the reviewer for highlighting the practical availability of contextual metadata. It is important to assess robustness to missing or unreliable phase and topic annotations in free-form group conversations and to distinguish metadata used during retrieval from metadata provided to the reader. Following your suggestion, we evaluated a variant that removes phase and topic annotations and additionally excludes role metadata. These fields are removed throughout graph construction, retrieval, and evidence presentation to the reader, while authorship information is retained.

MethodAverageMulti-HopUpdateAmbiguityImplicitTemporalAbstention
DG-MEM55.3052.2043.0032.0857.1453.0988.49
No phase/topic/role52.7548.9039.2521.7057.1452.4790.65
Hindsight46.0142.3117.7637.7440.8254.9477.06
SimpleMem28.7225.828.4112.2620.4127.7864.75
MemGPT28.1522.5317.7620.7528.5712.4277.98

Without role, phase, and topic metadata, DG-Mem achieves 52.75% overall accuracy, a decrease of 2.55 percentage points from the full model, retaining 95.4% of its accuracy. This variant still exceeds Hindsight, the strongest baseline reported in Table 1, by 6.74 percentage points. These results show that DG-Mem retains a substantial overall advantage on GroupMemBench without requiring these contextual annotations. We have added this experiment and the accompanying discussion to Section 4.6 of the revised version.

W2. Retrieval and query-to-answer efficiency.

We measured the full online pipeline, from receiving a query to retrieving evidence and generating the final answer, on all questions in GroupMemBench. Experiments ran on a Linux server equipped with an NVIDIA A10 GPU. After 10 warm-up queries, we collected measurements using one CPU thread and processing one query at a time. The table reports mean latency in seconds per query and accuracy in percent.

MethodRetrieval mean (s)QA mean (s)End-to-end mean (s)Accuracy
bm250.216.927.143.22
text-embedding-3-large1.426.768.238.04
LightMem0.248.959.216.78
SimpleMem40.197.0047.228.72
Hindsight2.6381.2083.846.01
DG-MEM1.236.88.055.30

DG-Mem retrieves evidence in 1.23 seconds per query and completes the full query-to-answer pipeline in 8.0 seconds, only 0.9 seconds more than BM25, the fastest baseline. Compared with Hindsight, the strongest baseline in accuracy, DG-Mem reduces mean end-to-end latency from 83.8 to 8.0 seconds, a 10.5-fold speedup. Retrieval accounts for approximately 15% of DG-Mem's total latency.

W3. Presentation of Tables.

We thank the reviewer for this helpful suggestion. In the revised version, we highlight the best result in bold and underline the second-best result in each column. We also report performance differences relative to the strongest baseline in each column, including negative differences where DG-Mem does not outperform it. These changes have been applied consistently to Tables 1–2 and Tables 6–13 in the appendix.


Thank you for your expertise, efforts, and thoughtful feedback. We have revised the PDF according to your constructive suggestions. Please let us know if you have further questions or suggestions.

Thank you again for your time and thoughtful review. With only three days remaining in the discussion period, we would greatly appreciate your feedback on whether our rebuttal has addressed your concerns. If any concerns remain, we would be happy to provide further clarification.

I thank the authors for partially addressing my concerns. However, the additional comparisons do not appear to cover all the baselines reported in the main paper (without further explanation), and the results do not include query-time token consumption. The incomplete reporting makes it difficult to rule out the possibility of selective reporting and to fully assess the effectiveness and efficiency of the proposed method. Therefore, I will maintain my original rating.

Reviewer_gm4zOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DG-MEM represents conversational history through a content-centric graph and a user-context graph connected by shared message anchors. Content retrieval seeds asker-conditioned propagation in the second graph, followed by query-conditioned candidate refinement. Construction and retrieval use local encoders and deterministic operations without generative LLM calls. The method achieves 55.30% accuracy on GroupMemBench and improves average F1 and BLEU-1 on LoCoMo across two answer models. Further experiments examine backbone robustness, component ablations, and construction efficiency.

Critical strengths
  1. Explicit treatment of asker-dependent relevance. The representation preserves authorship and interaction context while allowing retrieval across participants, addressing a practical limitation of content-only matching.
  2. Consistent empirical benefits with component evidence. Gains extend across GroupMemBench, LoCoMo, and several answer models. Separate ablations support both graphs and the conditioning/refinement stages, while construction measurements demonstrate substantial savings against the evaluated generative memory systems.
Critical weaknesses
  1. The need for two separate graphs remains insufficiently established. Removing either graph demonstrates that its information and processing are useful, but also removes relations and retrieval signals. A single heterogeneous graph containing the same nodes and relations, with comparable asker conditioning and reranking, would distinguish the benefit of separating graph views from the benefit of adding user-context information.
  2. Evidence for general multi-user effectiveness and efficiency is limited. GroupMemBench is the only multi-user benchmark, and the method uses contextual fields such as roles, channels, phases, and topics. Robustness to missing or noisy metadata is not evaluated. Construction savings are convincing against the selected generative baselines, but query-time latency and scaling are unreported despite two propagation stages and cross-encoder refinement.
Questions for authors
  1. How does DG-MEM compare with a single heterogeneous graph preserving the same relations, asker information, and refinement procedure?
  2. How much performance is retained when role, phase, or topic metadata is unavailable or noisy?
  3. What are retrieval latency and memory usage as history grows, compared with strong hybrid-retrieval and graph baselines?
4 comments
AuthorsOpenReview ↗

Response to Reviewer gm4z (1/2)

We sincerely thank the reviewer for the thorough evaluation and for highlighting our explicit treatment of asker-dependent relevance, the consistent gains across benchmarks and answer models, the supporting ablation results, and the substantial savings in memory construction. Below, we provide detailed and targeted responses to each of the concerns raised.

Q1. Compare with a single heterogeneous graph

We thank the reviewer for this important suggestion. It is important to distinguish the benefit of separating graph views from that of introducing user-context information. Following your suggestion, we evaluated a single heterogeneous graph that retains all nodes and relations from the original two graphs, and incorporates the same message information, asker conditioning, and query-conditioned candidate refinement mechanism. We kept the question set, reader, reranker, and final evidence budget consistent throughout the evaluation.

MethodAverageMulti-HopUpdateAmbiguityImplicitTemporalAbstention
DG-MEM55.3052.2043.0032.0857.1453.0988.49
Single heterogeneous graph51.6850.0041.1226.4251.0246.3087.77

DG-Mem achieves 55.30% overall accuracy, compared with 51.68% for the single heterogeneous graph, a gain of 3.62 percentage points. Improvements occur across all six categories, including 5.66 points on Ambiguity, 6.12 on Implicit, and 6.79 on Temporal.

This comparison validates the value of the dual-view retrieval design. In DG-Mem, content-based propagation first identifies query-relevant evidence, whose ranking initializes propagation in the user-context graph through aligned message anchors. The second stage is then conditioned on the asker and query-matched contexts. This separation gives content discovery and interaction-aware evidence selection distinct transition structures and personalization distributions, while allowing information to pass between the two views. The comparison therefore provides evidence for the value of separating and coordinating these retrieval processes beyond simply including user-context information. We have added this comparison result and relevant discussion in Section 4.6 of the revised version.

Q2. Performance without role, phase, and topic metadata.

We thank the reviewer for raising this concern. To assess robustness to missing contextual metadata, we jointly removed role, phase, and topic fields during graph construction, retrieval, and evidence presentation to the reader. We report the resulting performance below, alongside the original baseline results from Table 1 for reference.

MethodAverageMulti-HopUpdateAmbiguityImplicitTemporalAbstention
DG-MEM55.3052.2043.0032.0857.1453.0988.49
No phase/topic/role52.7548.9039.2521.7057.1452.4790.65
Hindsight46.0142.3117.7637.7440.8254.9477.06
SimpleMem28.7225.828.4112.2620.4127.7864.75
MemGPT28.1522.5317.7620.7528.5712.4277.98

Removing these three fields reduces overall accuracy from 55.30% to 52.75%, a decrease of 2.55 percentage points, retaining 95.4% of the full model's accuracy. The metadata-ablated variant still exceeds Hindsight, the strongest baseline in Table 1, by 6.74 percentage points. These results indicate that DG-Mem's overall advantage does not depend on the availability of complete role, phase, and topic annotations. We have added this comparison result and relevant discussion in Section 4.6 of the revised version.

AuthorsOpenReview ↗

Response to Reviewer gm4z (2/2)

Q3. Query-time latency, memory usage, and scaling.

We measured retrieval latency and memory usage using the first 5K, 15K, and 30K messages from the ordered Finance history. Since shorter histories may omit gold supporting evidence, we used these subsets only for efficiency measurements, not accuracy evaluation. All experiments ran on a Linux server with an NVIDIA A10 GPU, using one CPU thread and processing one query at a time. We ran 10 warm-up queries and repeated each measurement three times.

HistoryMethodMean retrieval (s/query)Peak RAM (GiB)Peak GPU allocated (GiB)
5KHindsight1.362.480.521
5KDG-MEM0.631.4640.478
15KHindsight1.823.060.522
15KDG-MEM0.871.6850.478
30KHindsight2.633.780.521
30KDG-MEM1.232.0110.478

DG-Mem has lower retrieval latency and peak memory usage than Hindsight at all three history sizes. Increasing the history sixfold, from 5K to 30K messages, raises DG-Mem's mean retrieval latency from 0.63 to 1.23 seconds per query, a 1.95-fold increase. Peak process RAM increases by 37.4%, while peak allocated GPU memory remains approximately 0.48 GiB.


Thank you again for your time, efforts, and thoughtful feedback. We have incorporated your helpful comments in the revised version. If you have any additional questions or suggestions, we would be happy to discuss them.

Thank you again for your time and thoughtful review. With only three days remaining in the discussion period, we would greatly appreciate your feedback on whether our rebuttal has addressed your concerns. If any concerns remain, we would be happy to provide further clarification.

Dear Reviewer gm4z,

With about ten hours left in the discussion period, we would appreciate your feedback on whether our response and additional results address your concerns. If any questions remain, we would be happy to clarify them before the discussion closes. Thank you again for your time.

Best wishes,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and reviewers for their time and constructive feedback. To facilitate the Area Chair’s assessment, we summarize the recognized strengths and key outcomes of our rebuttal.

The reviewers highlighted several contributions:

  • Important and underexplored problem. Shared assistants require memory that preserves user-specific context and distinguishes content relevance from asker relevance. [sQhJ, 2CFk, DXnd]
  • Well-motivated and intuitive design. The complementary graphs enable cross-user retrieval while preserving attribution and supporting traceable evidence selection. [All reviewers]
  • Strong empirical evidence. Reviewers recognized the promising results, consistent gains across benchmarks and answering models, and supporting ablations. [gm4z, mg8H, 2CFk]
  • Practical efficiency. Memory construction and retrieval require no generative-LLM calls, substantially reducing overhead and facilitating deployment. [All reviewers]

Our rebuttal strengthened these contributions through additional experiments and clarifications:

  • sQhJ: Main concern resolved, rating increased from 2 to 3. We added metadata-removal experiments, complete latency and token comparisons across all GroupMemBench baselines, and an explanation of the design choices underlying our efficiency. The reviewer confirmed that the results addressed their main concern and raised their rating, explicitly recognizing the novelty of both the problem formulation and the method.
  • 2CFk: No further questions. We provided hyperparameter sensitivity experiments and a breakdown of NER and embedding overhead. Performance remained strong across tested configurations. The reviewer confirmed that they had no further questions.
  • DXnd: Most concerns explicitly addressed. We clarified metadata availability, indirect-reference retrieval, refinement rules and fallbacks, and the LoCoMo configuration. The reviewer confirmed that our clarifications and experiments addressed most concerns.
  • mg8H: Positive acknowledgment followed by additional evaluation checks. The reviewer acknowledged the candidate-budget clarification and stronger evidence from the single-graph control. In response to subsequent questions, we evaluated the corrected 715-question set (updated by the benchmark maintainers on November 4, 40 days after our ICLR submission) with the benchmark reader prompt, obtaining 53.29% accuracy versus Hindsight’s 46.01%. We believe these results address the remaining concerns well; no further reply followed.
  • gm4z: Necessity of two graphs, metadata robustness, and retrieval scalability. We added a controlled single heterogeneous graph retaining the same nodes, relations, asker information, and refinement procedure. DG-Mem achieved 55.30% versus 51.68% accuracy, supporting the value of separating and coordinating the two graph views. Removing role, phase, and topic metadata retained 52.75% accuracy on the original evaluation set. We further measured latency and memory usage from 5K to 30K messages. We believe this evidence substantially addresses the concerns; the reviewer did not provide a follow-up during the discussion.

DG-Mem combines a important multi-user setting, a distinctive dual-graph retrieval design, and strong effectiveness with low computational cost. We respectfully ask the Area Chair to consider these contributions and the positive discussion outcomes when forming your recommendation.

Sincerely,

Authors of Paper #37056

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.