- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces BetweenUsBench, a benchmark designed to evaluate dyad-specific memory in long-term conversational agents. The benchmark assesses 2 dimensions: idiosyncratic reference, which measures the ability to interpret user-specific expressions and meanings, and interactional rapport, which evaluates whether agents can respond according to preferences implicitly learned from past conversations. Using synthetically generated multi-session conversations, the authors evaluate 7 memory methods across three LLM backbones and analyze failures in memory preservation, retrieval, andutilization. The results reveal limitations in existing memory systems, even when relevant information is present in conversation histories, highlighting the need for better mechanisms to retrieve and apply user-specific conversational knowledge.
Critical strengths
Underexplored research direction. The paper addresses an important and relatively underexplored aspect of long-term conversation, particularly relevant to AI companions.
Systematic dataset construction. The synthetic data generation process supports systematic construction of challenging cases with controlled evidence and response requirements, offering potential scalability.
Diagnostic evaluation. The evaluation framework examines memory preservation, retrieval, and utilization, providing diagnostic insights into current memory systems.
Critical weaknesses
Insufficient validation of real-world validity. Although the authors conduct human audits and report high agreement between LLM judges and human annotators, it remains unclear how well the benchmark reflects real-world conversational dynamics and user-specific interaction patterns.
Limited analysis. The paper mainly reports overall performance, with limited analysis of which types of cases are particularly challenging. More fine-grained analyses, such as performance across different challenge categories (e.g., boundary avoidance, ritual enactment) and common failure patterns would help clarify what makes the benchmark difficult and what capabilities current models lack.
Limited exploration of prompting-based baselines. It remains unclear whether the observed failures stem from memory limitations or could be addressed through simpler prompting strategies. For example, explicitly prompting models to identify relevant evidence or reflect on user-specific patterns before responding could help clarify whether these challenges primarily arise from memory mechanisms or inference-time reasoning.
Questions for authors
What memory mechanisms would be most effective for the capabilities evaluated in BetweenUsBench? Have the authors considered whether optimizing memory systems for these capabilities might introduce trade-offs with traditional task-oriented memory performance, particularly under limited memory budgets?