- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DyadMem evaluates user memory and how an agent should work with a particular user. It introduces User-conditioned Relational Agent Memory alongside user-side and episodic memory, with capture, update, and query-support annotations over synthetic dialogues. Separate stage evaluations and gold-memory versus full-pipeline QA expose failures hidden by answer accuracy alone. Experiments across diverse models reveal incomplete capture and retrieval, unreliable deletions, and answering-style effects. A controlled comparison tests user-conditioned memory construction.
Critical strengths
-
A diagnostic benchmark with detailed lifecycle supervision. Session-level labels, update operations, and query-to-memory supports enable evaluation of distinct memory capabilities. Gold-memory and full-pipeline settings reveal performance gaps, while grounded accuracy and abstention analysis expose limitations of answer-only evaluation.
-
Substantial annotation and evaluation effort. Large-scale dialogue construction, human review, and provenance checks support benchmark quality. Broad model coverage, cross-judge comparisons, and human calibration strengthen evaluation reliability, while released prompts, metrics, and code facilitate follow-up research.
Critical weaknesses
-
The distinct value of relational memory remains unclear. Although both variants share the same conversations, the Agent-only prompt excludes user-conditioned information targeted by evaluation, potentially favoring relational memory. A conventional profile-plus-episodic baseline or histories with multiple agents serving one user would better establish its unique contribution.
-
Stage-level diagnoses need stronger validation. Human calibration does not cover the embedding and entailment thresholds used for memory matching, complicating under-extraction claims. Excluding no-change sessions under-tests unnecessary updates, while fully answerable QA prevents evaluation of abstention when evidence is unavailable.
Questions for authors
-
Compare URAM with a user-profile-plus-episodic baseline preserving equivalent information under a matched memory budget, without relational type labels.
-
Evaluate paired histories involving the same user but different agents to establish whether target memories are relationship-specific rather than user-profile information.
-
Report human agreement and threshold sensitivity for capture and update matching across memory granularities, along with incorrect edit rates on no-change sessions.
-
Reconcile protocol descriptions and report episode-clustered uncertainty for QA comparisons sharing a memory bank.
5 comments
Response (2/2): Clarifications on URAM properties and stage‑wise benchmark validation
Dear Reviewer bsp5:
Thank you for your careful assessment and constructive queries. Your observations motivate us to further validate and elaborate on the core properties of URAM and our benchmark protocols.
Q1: Compare URAM with a user-profile-plus-episodic baseline preserving equivalent information under a matched memory budget, without relational type labels.
Response. Scoring is neutral to type labels. Capture and Update additions use semantic coverage matching, and the main F1 does not require matching types. Recall is scored by index on a shared gold bank, and QA reads only the answer content. A system that stores the same content as profile plus episodic loses nothing in extraction or QA for lacking URAM labels that labels are diagnostic only.
Content and organization can be tested separately. The controlled comparison in §4.3 Table 3 shares the same episodic memory and recall budget. The only difference is whether agent-side memory construction sees user-side memory. This is the requested baseline (see W1).
Q2: Evaluate paired histories involving the same user but different agents to establish whether the target memory is relationship-specific rather than user-profile information.
Response. It is relationship-specific. Gold labels define scope at three levels. Relation: each trajectory is a separate user–agent collaboration with its own bank and gold. Role: different assistant roles in one relation have separate rule sets; in at least 274 trajectories one user has the assistant take two or more roles, and 3,037 QA items test whether the right one is used. Time: rules have validity intervals, and expired ones are removed by Update. A user-profile store cannot express any of the three. 76% of URAM entries carry an explicit condition cue.
Q3: Report human agreement and threshold sensitivity for capture and update matching across memory granularities, along with incorrect edit rates on no-change sessions.
Response. Annotation agreement is that primary and second-pass judgments agree on 96% of the 218 overlapping episodes (Cohen's κ = 0.81), adjudication corrected 5.5% and removed 1.83%. Thresholds and no-change sessions are addressed in W2 (a) and (b).
Q4: Reconcile protocol descriptions and report episode-clustered uncertainty for QA comparisons sharing a memory bank.
Response. All models use the same prompt version, the same judge (GPT-5-mini), the same matching thresholds and NLI configuration, and the same scoring rules. Clustering by episode treats questions that share a memory bank as dependent. The resampling unit is 3,065 trajectories, not 61,210 questions, and paired comparisons are aligned within an episode. For Qwen3.5-9B, Full-Pipeline is 38.08% (question-level 95% CI 37.69–38.45 and episode-clustered 37.48–38.68) and Gold-Memory is 91.25% (91.02–91.47 and 91.01–91.49). The intervals widen slightly. Moreover, the conclusions and ranking do not change.
Thank you for the author's response; most of my concerns have been addressed. Please incorporate the corresponding supplementary experiments and relevant analyses into the revised version of the paper. Once these updates are made, I will consider adjusting my score accordingly.
Response (1/2): Clarifications on URAM properties and stage‑wise benchmark validation
Dear Reviewer bsp5:
We sincerely appreciate your thoughtful critique and detailed suggestions, which help us better clarify the core motivation and boundary of our URAM formulation.
Response. We agree and will narrow §4.3. Table 3 shows that cross-user agent memory covers user-conditioned rules less well (∆op-F1 > 0 for all 20 models, ∆QA > 0 for 17). Both arms read exactly the same user–assistant dialogues. Agent-only is not blind to user information. It is set up to record only agent knowledge that generalizes across users. This is the dominant practice in agent memory today, not a weakened control we built on purpose. Scoring it against gold that carries user-specific information is meant to test whether this general agent-memory paradigm can express personalized rules about how the assistant should work with the current user.
Both arms use the same session data, the same Capture and Update pipeline, and the same scoring rules. The only variable is which memory types may be stored. Any difference between the two arms can therefore come only from the memory type itself. Scoring does not favor URAM labels, because Capture and Update additions use semantic coverage matching, Recall is scored by index on a shared gold bank, and QA judges only whether the answer content is correct. The value of relational memory lies not in the label but in the content it carries. The collaboration rules in URAM come from specific interactions one by one that they cannot be derived from a static user profile alone.
Response. We make every QA item answerable so that an abstention in the Full-Pipeline can be attributed clearly to a failure in memory construction or memory retrieval rather than to a question with no answer. Whether the model abstains when evidence is missing has already been verified by controlled experiments (see Reviewer 1, Q1 and Q2): with the same model and the same prompt, over-abstention is 99.98% when no memory is injected, 97.96% when mismatched memory from the same domain is injected, and only 1.70% when correct gold memory is injected. When the relevant evidence is absent the model almost always abstains, which shows that its abstention behavior is highly sensitive to whether evidence is available. The 43.10% abstention in the Full-Pipeline is rooted in missing memory.