acceptodds
Under review as a conference paper at ICLR 2027

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

104 views

Abstract

Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history evolves. Existing benchmarks primarily supervise user facts and preferences or experience reusable across users, leaving this relationship-specific agent memory implicit. Additionally, most prior works measure the model solely with final-answer QA over long interaction histories, making the assessment still incomplete and unreliable. To this end, we introduce DyadMem with the proposed new definition User-conditioned Relational Agent Memory (URAM). DyadMem jointly annotates user-side memory and URAM along the same multi-session trajectories, resulting in 6 memory categories. To summarize, it includes 3,065 episodes, 50,961 sessions, and 61,210 QA instances, with extensive session-level Capture and Update gold annotations, query-level Recall support, and two QA settings: Gold-Memory and Full-Pipeline. Across 16 open-weight and 4 proprietary models, Gold-Memory QA is consistently strong, yet Full-Pipeline QA drops sharply. Such a gap explicitly supports our fine-grained evaluation design. Additionally, several quantitative results further reveal low Capture recall, incomplete Recall, and unsafe-deletion issues arising from even the frontier LLMs. We further conduct a rigorous experiment to validate the effectiveness of our URAM and observe the positive effects for all 20 models. In summary, DyadMem is a dual-domain, full-pipeline memory benchmark with extensive annotation efforts for advancing the domain's development.

Primary area:
datasets and benchmarks
Keywords:
Memory Benchmark, Long-term Memory, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bsp5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DyadMem evaluates user memory and how an agent should work with a particular user. It introduces User-conditioned Relational Agent Memory alongside user-side and episodic memory, with capture, update, and query-support annotations over synthetic dialogues. Separate stage evaluations and gold-memory versus full-pipeline QA expose failures hidden by answer accuracy alone. Experiments across diverse models reveal incomplete capture and retrieval, unreliable deletions, and answering-style effects. A controlled comparison tests user-conditioned memory construction.

Critical strengths
  1. A diagnostic benchmark with detailed lifecycle supervision. Session-level labels, update operations, and query-to-memory supports enable evaluation of distinct memory capabilities. Gold-memory and full-pipeline settings reveal performance gaps, while grounded accuracy and abstention analysis expose limitations of answer-only evaluation.

  2. Substantial annotation and evaluation effort. Large-scale dialogue construction, human review, and provenance checks support benchmark quality. Broad model coverage, cross-judge comparisons, and human calibration strengthen evaluation reliability, while released prompts, metrics, and code facilitate follow-up research.

Critical weaknesses
  1. The distinct value of relational memory remains unclear. Although both variants share the same conversations, the Agent-only prompt excludes user-conditioned information targeted by evaluation, potentially favoring relational memory. A conventional profile-plus-episodic baseline or histories with multiple agents serving one user would better establish its unique contribution.

  2. Stage-level diagnoses need stronger validation. Human calibration does not cover the embedding and entailment thresholds used for memory matching, complicating under-extraction claims. Excluding no-change sessions under-tests unnecessary updates, while fully answerable QA prevents evaluation of abstention when evidence is unavailable.

Questions for authors
  1. Compare URAM with a user-profile-plus-episodic baseline preserving equivalent information under a matched memory budget, without relational type labels.

  2. Evaluate paired histories involving the same user but different agents to establish whether target memories are relationship-specific rather than user-profile information.

  3. Report human agreement and threshold sensitivity for capture and update matching across memory granularities, along with incorrect edit rates on no-change sessions.

  4. Reconcile protocol descriptions and report episode-clustered uncertainty for QA comparisons sharing a memory bank.

5 comments
AuthorsOpenReview ↗

Response (1/2): Clarifications on URAM properties and stage‑wise benchmark validation

Dear Reviewer bsp5:

We sincerely appreciate your thoughtful critique and detailed suggestions, which help us better clarify the core motivation and boundary of our URAM formulation.

W1: The distinct value of relational memory is unclear, the Agent-only prompt excludes the user-conditioned information targeted by evaluation and may favor relational memory.

Response. We agree and will narrow §4.3. Table 3 shows that cross-user agent memory covers user-conditioned rules less well (∆op-F1 > 0 for all 20 models, ∆QA > 0 for 17). Both arms read exactly the same user–assistant dialogues. Agent-only is not blind to user information. It is set up to record only agent knowledge that generalizes across users. This is the dominant practice in agent memory today, not a weakened control we built on purpose. Scoring it against gold that carries user-specific information is meant to test whether this general agent-memory paradigm can express personalized rules about how the assistant should work with the current user.

Both arms use the same session data, the same Capture and Update pipeline, and the same scoring rules. The only variable is which memory types may be stored. Any difference between the two arms can therefore come only from the memory type itself. Scoring does not favor URAM labels, because Capture and Update additions use semantic coverage matching, Recall is scored by index on a shared gold bank, and QA judges only whether the answer content is correct. The value of relational memory lies not in the label but in the content it carries. The collaboration rules in URAM come from specific interactions one by one that they cannot be derived from a static user profile alone.

W2: Stage-level diagnoses need stronger validation. Human calibration does not cover the matching thresholds; excluding no-change sessions under-tests unnecessary edits; fully answerable QA leaves abstention under missing evidence untested.

Response. We make every QA item answerable so that an abstention in the Full-Pipeline can be attributed clearly to a failure in memory construction or memory retrieval rather than to a question with no answer. Whether the model abstains when evidence is missing has already been verified by controlled experiments (see Reviewer 1, Q1 and Q2): with the same model and the same prompt, over-abstention is 99.98% when no memory is injected, 97.96% when mismatched memory from the same domain is injected, and only 1.70% when correct gold memory is injected. When the relevant evidence is absent the model almost always abstains, which shows that its abstention behavior is highly sensitive to whether evidence is available. The 43.10% abstention in the Full-Pipeline is rooted in missing memory.

AuthorsOpenReview ↗

Response (2/2): Clarifications on URAM properties and stage‑wise benchmark validation

Dear Reviewer bsp5:

Thank you for your careful assessment and constructive queries. Your observations motivate us to further validate and elaborate on the core properties of URAM and our benchmark protocols.

Q1: Compare URAM with a user-profile-plus-episodic baseline preserving equivalent information under a matched memory budget, without relational type labels.

Response. Scoring is neutral to type labels. Capture and Update additions use semantic coverage matching, and the main F1 does not require matching types. Recall is scored by index on a shared gold bank, and QA reads only the answer content. A system that stores the same content as profile plus episodic loses nothing in extraction or QA for lacking URAM labels that labels are diagnostic only.

Content and organization can be tested separately. The controlled comparison in §4.3 Table 3 shares the same episodic memory and recall budget. The only difference is whether agent-side memory construction sees user-side memory. This is the requested baseline (see W1).

Q2: Evaluate paired histories involving the same user but different agents to establish whether the target memory is relationship-specific rather than user-profile information.

Response. It is relationship-specific. Gold labels define scope at three levels. Relation: each trajectory is a separate user–agent collaboration with its own bank and gold. Role: different assistant roles in one relation have separate rule sets; in at least 274 trajectories one user has the assistant take two or more roles, and 3,037 QA items test whether the right one is used. Time: rules have validity intervals, and expired ones are removed by Update. A user-profile store cannot express any of the three. 76% of URAM entries carry an explicit condition cue.

Q3: Report human agreement and threshold sensitivity for capture and update matching across memory granularities, along with incorrect edit rates on no-change sessions.

Response. Annotation agreement is that primary and second-pass judgments agree on 96% of the 218 overlapping episodes (Cohen's κ = 0.81), adjudication corrected 5.5% and removed 1.83%. Thresholds and no-change sessions are addressed in W2 (a) and (b).

Q4: Reconcile protocol descriptions and report episode-clustered uncertainty for QA comparisons sharing a memory bank.

Response. All models use the same prompt version, the same judge (GPT-5-mini), the same matching thresholds and NLI configuration, and the same scoring rules. Clustering by episode treats questions that share a memory bank as dependent. The resampling unit is 3,065 trajectories, not 61,210 questions, and paired comparisons are aligned within an episode. For Qwen3.5-9B, Full-Pipeline is 38.08% (question-level 95% CI 37.69–38.45 and episode-clustered 37.48–38.68) and Gold-Memory is 91.25% (91.02–91.47 and 91.01–91.49). The intervals widen slightly. Moreover, the conclusions and ranking do not change.

Thank you for the author's response; most of my concerns have been addressed. Please incorporate the corresponding supplementary experiments and relevant analyses into the revised version of the paper. Once these updates are made, I will consider adjusting my score accordingly.

Reviewer_YPywOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces DyadMem, a large-scale benchmark for evaluating long-term agent memory across user-side and agent-side memory domains. Its central concept, User-conditioned Relational Agent Memory (URAM), represents how a particular agent should work with a particular user based on their interaction history. The benchmark provides session-level Capture and Update annotations, query-level Recall supports, and two QA settings that distinguish answer generation with gold memories from end-to-end memory-pipeline performance. The authors evaluate 20 open-weight and proprietary models and report substantial gaps between Gold-Memory QA and Full-Pipeline QA, along with weaknesses in memory capture, updating, and retrieval.

Critical strengths
  1. The authors report human calibration of the QA judge, cross-judge ranking consistency, and bootstrap confidence intervals. These are valuable steps toward a reliable benchmark.
  2. DyadMem contains 3,065 episodes, 50,961 sessions, and 61,210 QA instances. The paper evaluates 20 models and reports several complementary metrics, including grounded accuracy, over-abstention, and unsupported correct answers.
Critical weaknesses
  1. The authors state that "URAM is not another description of the user: two agents serving the same user may require different future behavior because their interaction histories differ". However, this observation is not sufficient to establish URAM as a distinct memory abstraction. Different agents naturally accumulate different interaction trajectories and may consequently develop different representations of the same user. When agents retrieve relevant information from their respective memory stores, these differences can already lead to different behaviors without requiring a separate relational memory category. From a conditional-probability perspective, an agent's behavior is already conditioned on user-specific information when user memory is available at inference time. Furthermore, the example in Figure 1 raises concerns about the redundancy of URAM. The user memory contains "This user loves spicy food", while the agent memory contains "Recommend spicy food first". The latter appears largely derivable from the former, given the agent's access to the user memory and its parametric commonsense knowledge. This raises the question of what additional information URAM captures that cannot be inferred from user memory and the agent's existing capabilities, and whether maintaining a separate memory is necessary. If agent-side memory is instead intended to capture persistent skill acquisition and procedural learning, as in systems such as Hermes that use MEMORY.md to retain learned skills, then the paper's motivation and examples appear misaligned with that interpretation.
  2. Although DyadMem evaluates 20 LLMs, its evaluation primarily varies the backbone models rather than the underlying memory systems. For a benchmark intended to evaluate long-term memory, testing diverse memory architectures and implementations is important for establishing its generality and practical usefulness. Different systems may employ substantially different memory representations and update mechanisms, and strong results across many backbones do not establish that the benchmark can fairly evaluate these systems. One possible reason for this limitation is the benchmark's emphasis on fine-grained lifecycle metrics, such as Add F1 and Deletion F1, which may require system-specific adaptation when memory entries, updates, and deletions are represented differently across implementations. If so, this design creates a tension between detailed process-level evaluation and broad applicability: integrating a new memory system may require additional adapters or instrumentation, potentially limiting adoption by memory-system developers. The authors should evaluate DyadMem on several representative existing memory systems, or at least provide a clear and general integration protocol that explains how systems with different internal representations and update mechanisms can be evaluated fairly. They should also clarify which metrics can be applied in a system-agnostic manner and which require access to system-specific operations.
  3. More broadly, the paper can be viewed as evaluating personalization and continual learning simultaneously on the same interaction trajectories. This framing appears incremental unless the proposed distinction between user-side and agent-side memory reveals unique failure modes or enables capabilities that existing personalized-memory approaches cannot provide. However, the paper does not convincingly demonstrate such distinctive failure modes or establish the necessity of URAM as a separate memory abstraction.
Questions for authors

I have no other questions.

4 comments
AuthorsOpenReview ↗

URAM motivation, memory‑system experiments and contribution clarifications

Dear Reviewer YPyw

Thank you for your critical review. Your thoughtful observations push us to further justify the motivation behind URAM, validate benchmark generality, and sharpen our statement of contributions.

W1:The necessity and redundancy of User‑conditioned Relational Agent Memory (URAM)

Response. Firstly, we would like to stress that the contribution is the benchmark itself. URAM is one of its supervision targets. Previous benchmarks only implicitly captured the knowledge of how an assistant should adapt to a specific user. URAM is not a novel memory storage design. Instead, it surfaces these adaptive rules explicitly so they can be grounded, verified and evaluated.

Secondly, regarding the Figure 1 example, "user likes spicy, so recommend spicy" is only a simplified illustration. It is true that a simple preference like this can be reconstructed from basic memory plus commonsense reasoning. But URAM in DyadMem dataset is not built on that kind of easy inference. In fact, 76% of our URAM entries carry explicit conditional cues (for/when/if/during/unless), grounded in concrete interactions. These conditional rules cannot be deduced merely from a static user profile.

Thirdly, incorporating user-side memory into the agent’s own memory evolution is better suited for capturing implicit user preferences. Your mentioned conditional-probability perspective assumes user memory is recalled and used at inference. However, Memory recall is selective and only a small subset of relevant memories are retrieved for each query, which means useful user information often stays unaccessed in the memory bank. Our URAM design shifts this work earlier to the memory update stage. While the agent organizes its own memory, it compiles user requirements into actionable rules. Subsequent tasks can directly rely on these rules instead of re‑running retrieval. In §4.3 Table 3, Update improves for all 20 models and QA for 17 (open-weight +3.2%, proprietary +3.7%, DeepSeek-V4-Pro +9.8%). This comparison is only possible with session-level URAM ground truth. Final QA performance alone cannot tell whether user information has actually been written into the agent’s memory.

Finally, the difference from Hermes is structural. Hermes puts memory into static files like MEMORY.md, which are injected in full and remain unchanged within a session, its content is mostly cross-user procedural skills. DyadMem is a benchmark for entry-level memory that is retrieved semantically and revised across sessions, and for training agents that learn to write memory rules.

W2:Memory-system evaluation

Response. We added further experiments to test six memory systems on the full DyadMem dataset, with Qwen3.5-9B and GPT-4o-mini as two separate answerers. QA accuracy differs by as much as 26 points for one answerer, and the ranking across memory systems remains stable no matter which answerer we use. This shows DyadMem is capable of telling memory systems apart, and validates the core value of our benchmark.

SystemQA (Qwen3.5-9B)QA (GPT-4o-mini)F1 (Qwen3.5-9B)F1 (GPT-4o-mini)Precision@16Recall@16
A-Mem74.8%72.2%0.2900.3200.1360.522
Mem073.0%69.7%0.2770.3020.1440.545
LangMem63.2%59.6%0.2380.2620.0250.086
MemOS55.6%52.8%0.2360.2630.0790.215
LightMem50.5%45.8%0.2530.2680.0700.263
MemoryOS48.4%45.0%0.2490.2710.0820.279

W3: Contributions of DyadMem

Response. We respectfully agree that combining personalization and continual learning is not a novel idea. However, our core contribution is not a novel framework or a brand-new memory system, but three concrete outputs.

  1. A dual-domain, long-context benchmark. We build it with 3,065 episodes, 50,961 sessions, and 61,210 QA pairs, covering both user and agent memory across long interaction trajectories. It addresses a clear gap in existing memory evaluations, which lacks dual domain and long context coverage, and we release it as reusable dataset for the agent memory community.
  2. Empirical evidence of capability gaps within the external vector memory paradigm. Models show substantial variation in how well they capture, update, and recall memory. Rather than just proposing an abstract framework, our quantitative results lay out empirical ground to analyze the bottlenecks and inherent trade-offs of this paradigm.
  3. A supervision target tailored for implicit user preferences. We find that inputting user-side memory into the agent’s own memory evolution during the update stage helps handle implicit user preferences better for tasks requiring agent-side memory. This is a design choice at the supervision target level. It conditions memory bank updates on the user’s implicit requirements. Importantly and respectfully, we do not argue that URAM is a universally required memory category.

Thank you for your response. However, I am unable to raise my score for the following reasons:

  • Hermes's MEMORY.md already supports entry-level memory management. Unlike URAM, as described by the authors, all entries in MEMORY.md are included in the prompt. Hermes supports adding, deleting, and modifying entries in MEMORY.md, as well as consolidating them. These operations can have effects that persist across sessions.

  • The authors emphasize that URAM's memory entries carry explicit conditional cues (for/when/if/during/unless). However, similar mechanisms have already been explored in prior work. For example, ReMe [1] introduces a "when_to_use" field to indicate when a particular memory would be useful. This further reinforces my view that the proposed work is largely incremental.

  • The memory system evaluation table you provided does not include the new metrics proposed in your paper. This also suggests that these metrics are applicable only within the specific memory system framework introduced in this work. At a minimum, I believe the authors should clarify how the proposed metrics can be applied to the existing memory systems included in the evaluation.

  • Regarding Table 1 in this paper, the three features, Per-session gold memory, Gold lifecycle annotations, and Query-to-bank, each introduce additional requirements that reduce the usability of a benchmark and make it more difficult to evaluate new memory systems. This limitation has already been highlighted in prior work. For example, ProMem [2] points out that MemoryOS and A-MEM cannot be evaluated using certain metrics provided by HaluMem.

[1] Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution

[2] Beyond Static Summarization: Proactive Memory Extraction for LLM Agents

AuthorsOpenReview ↗

Conceptual clarifications and benchmark generality

Dear Reviewer YPyw,

We sincerely appreciate for your critical and thoughtful feedback. Your comments helped us refine our core motivation and clarify important distinctions relative to previous published work.

Hermes comparison

Response. Thank you for your comment. We admit our earlier description was imprecise. Hermes [1] does support entry‑level memory operations. It can add, replace and delete entries inside MEMORY.md, with changes persisting across sessions. We would like to highlight a key distinction that Hermes is a memory‑system implementation, whereas DyadMem is an evaluation benchmark. DyadMem supplies annotated supervision signals to diagnose memory‑model pipeline instead of competing as an alternative memory system against Hermes.

  1. No native ground‑truth supervision in Hermes. Hermes lets the model itself decide when to add, modify or delete memories. It provides no ground truth specifying which entries should be written in each turn or which entries are obsolete. However, DyadMem fills this gap by offering session‑level Capture and Update ground truth along with query‑level supporting evidence. It can evaluate whether memory systems like Hermes perform correct read‑write behaviors, instead of replacing them.

  2. No independent recall stage in Hermes. Hermes connects the full MEMORY.md (2200‑character limit) directly into prompts without separate retrieval. By comparison, one DyadMem trajectory spans 16.6 sessions and contains 341.5 annotated memory entries on average which demands explicit recall. As shown in Table 2, recall failure is a major bottleneck for long‑term memory. This failure mode cannot occur under Hermes’s inherent design.

  3. Hermes storage is different from URAM. Hermes’s user store holds user‑profile information and its memory store keeps environment facts, project agreements and general tool‑related notes. It has no dedicated storage and no annotated supervision for relationship‑specific rules that what actions this assistant should take for this particular user under given scenarios. The core contribution of URAM is to formalize these behavioral rules as annotatable, verifiable supervision targets.

ReMe conditional memory

Response. Thank you for pointing out the related work ReMe. While ReMe’s when_to_use field marks relevant scenarios for memory lookup, there are critical conceptual differences from URAM.

ReMe’s [2] when_to_use purely serves as a retrieval‑oriented index. It tells the system when this memory should be fetched for vector‑based lookup. It only supervises memory retrieval, and does not annotate or validate what concrete action should be taken once that condition is satisfied.

However, condition keywords (for/when/if/during/unless) in URAM are just natural language inside free‑text memory content. We do not implement dedicated structured fields or special condition‑matching logic for memory recall. Retrieval fully relies on the model’s general contextual and question understanding. These conditional phrases do not receive special index processing.

As a benchmark, DyadMem enables us to evaluate condition‑mismatch failure cases. Scenarios where the correct memory is successfully retrieved, yet the agent fails to apply the associated conditional behavior. This failure mode sits outside ReMe’s evaluation scope. While our work builds upon previous thinking around conditional memory, DyadMem adds new evaluative capabilities that have not been covered in existing work.

Benchmark metrics generality

Response. Thank you for raising this practical limitation and citing ProMem. We agree that the full set of gold annotations relatively increases the barrier for complete end‑to‑end assessment. This is an inherent trade‑off for fine‑grained full‑lifecycle benchmarking, consistent with observations from ProMem about HaluMem. However, DyadMem supports layered adoption. External memory systems do not need to satisfy all three annotation requirements at once:

[1] Scenarios 1: Query‑to‑bank supports only. Evaluate QA + Recall only. No capture or update adaptation is required. The system merely outputs retrieved memories and final answers.

[2] Scenarios 2: Add per‑session gold memory. If the system outputs per‑session extracted memory entries, Capture metrics become available, with no need to expose internal update traces.

[3] Scenarios 3: Full evaluation with all three annotations. To get complete stage‑level Update diagnostics, the system must expose per‑session add / delete / edit operation traces.

Response Reference

[1] Hermes-agent

[2] Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution

[3] Beyond Static Summarization: Proactive Memory Extraction for LLM Agents

Thank you for your response. As the public discussion period is coming to an end, I have no further questions. I have raised my score, but I still lean toward rejecting this paper.

Reviewer_HaAXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes User-conditioned Relational Agent Memory (URAM) which is a memory storing how information on particular agent should work with this particular user (behavioral requirements, reusable procedures, and learned decision rules) and this paper builds a benchmark that annotates this memory and another memory (what is true about the user i.e user profiles, factual information, and shared episodic memories) on the same trajectories.

The dataset contains 3,065 episodes, 50,961 sessions, and 61,210 questions across five domains and 25 subdomains. It provides supervision for extracting memories from a session, adding, replacing, or deleting memories, selecting supporting entries from a memory bank and answering either from gold supporting memories or from a model-constructed memory.

20 LLMs were evaluated, with reasoning disabled for every reasoning-capable model except MiniMax-M2.5. Experiments across 20 models show a substantial gap between answering with gold evidence and answering through the complete pipeline. Average accuracy falls from approximately 90.6% to 47.7% for open-weight models and 91.2% to 57.4% for proprietary models. The authors also report low extraction recall, incomplete retrieval, and frequent incorrect deletions.

Critical strengths
  1. The joint annotation of user information and agent collaboration rules along the same evolving trajectory creates opportunities to study whether changes in user circumstances lead to appropriate changes in agent behavior
  2. Session-level Capture and Update supervision with lineage is rare at this scale. It could support training memory writing agents.
Critical weaknesses
  1. The QA-generation prompt requires third-person questions about the stored record, such as which boundaries the agent was given but the paper's motivation is what the agent should do differently for this user and no task checks whether a model with its memory actually does so.
  2. All 20 models run inside one fixed prompt scaffold, so no system with its own write and update its memory/logic is tested
  3. Fig. 2 reports 12K–44K tokens per episode, and its component breakdown attributes 25.9M tokens to dialogue turns, ≈8.4K per episode (25.9M / 3,065). Even the longest episode fits in a 128K-token window, and the average dialogue fits in Qwen3’s 32K native window, so a full-context baseline is feasible for most evaluated models. One cannot tell whether the 34–43-point G2P gap reflects the difficulty of long-term memory or the cost of this particular Capture-Update-Recall scaffold
  4. The Agent-only prompt tells the model not to condition memories on the user or store user-specific information (App. K), yet it is scored against gold agent-side entries that are, by definition, conditioned on this user. As designed, it shows that a prompt aligned with the gold's definition matches the gold better, not that URAM improves the agent.
Questions for authors
  1. No-Memory results are missing in Table 4
  2. Ungrounded-correct rates reach 27.4% (Qwen3-4B), 25.0% (GPT-OSS-120B), and 23.8% (Qwen3-30B-A3B). A sizable share of credited answers is therefore not supported by the supplied memories, so guessability needs to be quantified.
  3. Episodes in which the same user works with two agents (or two roles) whose histories diverge, does URAM stays correctly scoped?
  4. Why was the Agent-only prompt forbidden to store user-specific content, when the gold agent-side targets are user-specific?
  5. Which model drafted the gold Capture, Update, and QA labels?
  6. What fraction of the drafted entries, operations, and QA items did humans edit or remove?
  7. What is the inter-annotator agreement on the doubly reviewed overlap?
  8. Do the conclusions about frontier models' Update failures still hold with reasoning enabled?
8 comments
AuthorsOpenReview ↗

Response (1/3): Clarifications on benchmark design and supplementary experimental results

Dear Reviewer HaAX,

Thank you very much for your thorough, careful review and for raising precise questions that strengthen our benchmark design and validation. We address each point and present new supporting experiments including No‑Memory baseline (Q1), a mismatched-memory guessability control (Q2), six external memory systems (W2), and inter-annotator agreement (Q7). We will revise text where our original wording was over‑stated.

W1. The QA asks about the stored record, not whether the agent actually behaves differently for this user.

Response. Thank you for your comment. The third-person form only fixes the grammatical subject. Agent-centric questions ask how the assistant should act for this user in a given situation, e.g., "When a vendor proposes a quick sync, how should the assistant decide between email and a call …?" (App. L, p. 111). The QA prompt forbids restating a stored entry and requires applying rules to new situations, keeping exceptions, or picking the current rule over superseded ones (App. E). In Full-Pipeline QA, the model answers from its own bank, so DyadMem does test whether a model, using its own memory, knows how to act for this user. We use evaluator-style questions because each has a verified reference answer and gold support, so errors can be scored and traced to Capture, Update, or Recall.

W2: All 20 models run inside one fixed prompt scaffold, so no system with its own write and update its memory/logic is tested.

Response. We understand your concern. We agree that results from a fixed scaffold alone do not show that DyadMem can evaluate real memory systems. We fixed the scaffold so that differences among the 20 models can be attributed to the models rather than to pipeline design. To address the concern directly, we ran six memory systems on the full benchmark (3,065 episodes, 61,210 questions). Each system uses its own write, update, and retrieval logic. This table also appears in our response to Reviewer YPyw.

SystemQA (Qwen3.5‑9B)QA (GPT‑4o‑mini)F1 (Qwen3.5‑9B)F1 (GPT‑4o‑mini)Precision@16Recall@16
A‑Mem74.8%72.2%0.2900.3200.1360.522
Mem073.0%69.7%0.2770.3020.1440.545
LangMem63.2%59.6%0.2380.2620.0250.086
MemOS55.6%52.8%0.2360.2630.0790.215
LightMem50.5%45.8%0.2530.2680.0700.263
MemoryOS48.4%45.0%0.2490.2710.0820.279

QA = Full-Pipeline accuracy.

As shown in the table, QA accuracy differs by 26–27 points across systems, and the six systems rank in the same order. This suggests that DyadMem can tell memory systems apart and that the ranking does not depend on one particular answerer. QA and retrieval metrics apply to any system that exposes what it retrieves. Capture and Update metrics also need the system to expose its write operations. The revision will add this integration protocol and the table.

W3: Does the G2P gap reflect long-term memory or the scaffold? A full-context baseline is feasible.

Response. Thank you for your comment. DyadMem is designed as a long‑context memory benchmark. The long context input is our significant contribution in this research domain. Specifically, our average dialogue is indeed 8.4K tokens, and the longest is about 44K, so a Qwen model's 262K standard window size can easily fit all of it. However, the metric of G2P quantifies the gap between Gold-Memory and Full-Pipeline, and in both settings the model only sees a short list of memories. The gap therefore arises from memory bank construction and memories recall quality, not context window constraints. We score Capture, Update, and Recall separately on gold inputs, so the loss can be traced to a specific stage. A full-context baseline tests whether the model can read the whole dialogue and answer. It does not test whether the model builds and maintains a memory bank. It therefore does not change what G2P measures.

W4: The Agent-only prompt forbids user-conditioned content but is scored against user-conditioned gold.

Response. In this question, we would like to clarify that the user memories are different from the user-turns. In our setting, Agent-only reads the same user–assistant dialogues as the other variant. It is asked to store general agent knowledge only, but nothing prevents it from reading the user turns. Scoring it against user conditioned gold asks whether this paradigm can represent the target rules at all. On both Update and QA it trails by a consistent margin, which suggests it cannot. Both variants run on identical sessions, share the same Capture and Update pipeline, and use the exact same scoring protocol. The only distinction is which memory types are permitted. Respectfully, the performance gap we see is not caused by misalignment between the prompt and our gold definition.

AuthorsOpenReview ↗

Response (2/3): Clarifications on benchmark design and supplementary experimental results

Dear Reviewer HaAX,

Thank you for raising this question. We have run these additional experiments to clarify our points.

Q1: No-Memory results are missing in Table 4.

Response. We ran the No-Memory condition to fill the empty row in Table 4. Qwen3.5-9B answers all 61,210 questions over 3,065 episodes with no memory injected, using only the question text and its own knowledge.

No-Memory (Qwen3.5-9B)QA accuracyOver-abstention (OA)
user-centric (30,468)0.003%99.997%
agent-centric (30,742)0.039%99.961%

Cap., Upd., and Rec. stay at blank with no memory since there is nothing to extract, update, or recall.

Q2: Ungrounded-correct rates reach 27.4% (Qwen3-4B), 25.0% (GPT-OSS-120B), and 23.8% (Qwen3-30B-A3B)... guessability needs to be quantified.

Response. Guessability is negligible. With the same model, the same 61,210 questions, the same answer prompt, and the same judge (GPT-5-mini), we replace only the injected memory, for each question we draw the same number of memories as its gold recall from other episodes, with no overlap with that question's gold.

Setting (Qwen3.5-9B, 61,210 questions)QA accuracyOver-abstention (OA)
No memory0.02%99.98%
Mismatched memory, cross-domain (same count)0.12%99.16%
Mismatched memory, same subdomain, different user (same count)0.41%97.96%
Correct gold support (reference, Table 2)91.25%1.70%
Full pipeline with self-built bank (reference, Table 2)38.08%43.10%

Ungrounded-correct is not a guess rate. The metric's denominator is positively scored answers, not all questions. For Qwen3-4B, 27.36% of scored answers are unsupported, which is 11.93% of all questions. For GPT-OSS-120B and Qwen3-30B-A3B the shares of all questions are 15.47% and 5.40%. These answers usually come from adjacent evidence in the injected memory, and the gold support set marks only the entries needed to answer, so a correct paraphrase can still be labelled unsupported. Correcting by 0.41% moves every model by less than 0.5 points and leaves the grouping and ranking unchanged.

Q3: Episodes in which the same user works with two agents (or two roles) whose histories diverge, does URAM stay correctly scoped?

Response. Yes, it does. Scope is built into the benchmark’s structure. Our gold labels enforce scoping at three distinct levels. [1] Relation: each trajectory represents an independent user–agent collaboration with its own dedicated memory bank and ground truth. [2] Role: within a single user–agent relation, different assistant roles maintain separate rule sets, and our QA checks whether the model pulls up the correct rule for the active role. [3] Time: every rule has its own validity window, and the Update task retires rules once they expire. Notably, 76% of URAM entries are conditional and only activate under specific circumstances, and our QA explicitly asks whether a rule applies to the current situation. We also include cases where different users give opposite requirements for the same task. If memory were shared across separate user–agent relations, these rules would inherently conflict.

Q4: Why was the Agent-only prompt forbidden to store user-specific content, when the gold agent-side targets are user-specific?

Response. Agent-only is the standard setup, not a baseline we made weak on purpose. General agent memory is shared across users, so it holds nothing specific to one user. If we let it store user-specific content, it stops being Agent-only and the comparison no longer tests anything.

The gold agent-side entries are user-specific, and that is exactly what we evlauted. Both variants see the same sessions and run through the same Capture and Update steps. The only difference is which memory types they may keep (p.61 Table 9). In §4.3 Table 3, controlled Update improves for all 20 models, and URAM-only QA improves for 17. The average gain is +3.2% for open-weight models, +3.7% for proprietary models, and +9.8% for DeepSeek-V4-Pro. Agent-side memory serves the user consistently only when user-side memory takes part in the update.

AuthorsOpenReview ↗

Response (3/3): Clarifications on benchmark design and supplementary experimental results

Dear Reviewer HaAX,

Q5: Which model drafted the gold Capture, Update, and QA labels?

Q6: What fraction of the drafted entries, operations, and QA items did humans edit or remove? (Joint response for Q5 and Q6)

Response. All dialogues were generated by GPT-5.4, then self-checked for its temporal, role, and logical consistency. Capture, Update, and QA drafts also came from GPT-5.4, but the model only drafted and flagged suspected errors. However, it was forbidden to write gold labels. Only annotators decided all gold (Appendix D.1, G.2). GPT-5-mini was used only to score QA, and GPT-5.5 only as a translation aid on the review website page. Annotators edited based on the raw dialogues. Human review is described in Appendix D. We estimate the total annotation and experiment cost exceeded $57,751.

Q7: What is the inter-annotator agreement on the doubly reviewed overlap?

Response. Agreement was built into the second-review design (Appendix D). Five primary annotators each covered one domain, and every episode had one primary reviewer. Episodes that passed the first annotation went into a second pool. Three reviewers each drew a random sample of 500 episodes, and the samples were designed to overlap. Every episode in the overlap was read independently by at least two reviewers and counted only once for coverage. This lets us compare the primary and second-pass judgments directly.

MetricResult
Unique overlapping episodes218
Raw agreement, primary vs. second pass96%
Cohen's κ0.81
After adjudicationcorrected 5.5%, removed 1.83%

Q8: Do the conclusions about frontier models' Update failures still hold with reasoning enabled?

Response. Yes it does. All 20 models have a wrong-deletion rate above 62% under both deletion settings. Moreover, MiniMax-M2.5 is the only model whose reasoning cannot be turned off. It still has 3.06% wrong deletions and an op-F1 of 23.43, above the open-weight average of 18.55. Therefore, reasoning does not change the conclusion. In the table, 11 local models use relatively conservative deletion, at most 10 per session, and 9 API models have no cap.

ModelDeletion settingReasoning modeWrong-deletion rateUpdate op-F1
GPT-OSS-120BConservativeOff62.55%23.88
Qwen3.8-27BConservativeOff73.31%30.18
Qwen3.5-27BConservativeOff80.96%23.79
Llama-3.1-70BConservativeNone81.42%19.03
Qwen3.5-9BConservativeOff88.48%16.39
Qwen3-4BConservativeOff91.90%12.88
Qwen3-8BConservativeOff92.49%12.04
Qwen3.5-35B-A3BConservativeOff94.72%13.52
Qwen3-30B-A3BConservativeOff94.94%10.06
Qwen3.5-4BConservativeOff95.32%9.62
Llama-3.1-8BConservativeNone98.51%4.44
Conservative group average (11)86.78%15.98
Kimi-K3UncappedOff67.77%31.69
Doubao-Seed-2.1-ProUncappedOff69.15%30.68
MiniMax-M2.5UncappedForced on73.06%23.43
GPT-5.6-SolUncappedOff78.96%25.54
DeepSeek-V4-ProUncappedOff81.03%22.48
GPT-4o-miniUncappedNone81.89%13.67
Gemini-3.1-ProUncappedOff82.21%27.99
Qwen3.5-397B-A17BUncappedOff83.75%22.31
GLM-5.2UncappedOff85.64%21.06
Uncapped group average (9)78.16%24.32

The conservative group averages 86.78%, the uncapped group 78.16%, and the best model, GPT-OSS-120B, still reaches 62.55%. The bottleneck is deciding which memory has been superseded, not the reasoning budget. Capping deletions and using stronger models do not change the error type.

Comments on the paper

1 comment

Dear Senior Area Chair, Area Chair, and Reviewers,

We sincerely appreciate the constructive feedback from all reviewers. All three reviewers positively recognized our annotation quality and stage‑wise diagnostic framework.

Several core concerns stemmed from mis-interpretations of our benchmark design. We have clarified these points and supplemented new supporting experiments, including evaluations across six external memory systems, and no‑memory and mismatched‑memory controls.

To briefly restate our core contributions:

  1. DyadMem is a diagnostic long‑context temporal benchmark, instead of a newly‑designed memory system. Unlike previous works that only report end‑to‑end QA scores, it provides session‑level Capture and Update ground‑truth with provenance tracking. This enables us to pinpoint memory failures in capture, update and recall separately, and offers supervision signals for memory‑writing agent training.

  2. We formalize URAM (User‑conditioned Relational Agent Memory). It characterizes how an assistant should collaborate with a specific user, complementary to static user profiles. URAM exposes previously implicit interaction‑derived rules into annotatable supervision targets. Controlled ablations show that incorporating user‑side information during agent memory evolution brings consistent gains across most evaluated models.

  3. Large‑scale high‑quality annotated dataset. DyadMem covers 3,065 trajectories, 50,961 sessions and 61,210 QA instances across 5 domains and 25 sub‑domains, built via rigorous multi‑round human annotation. We estimate the total annotation and experiment cost exceeded $57,751.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.