acceptodds
Withdrawn from ICLR 2027

BetweenUsBench: Resonance Requires Dyad-Specific Memory, Not Mere Recall

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-term conversational memory supports both task execution and companionship. Beyond retaining facts, companionship requires understanding a user’s private meanings and context-dependent response expectations, supporting a sense of resonance: this agent understands me. Retaining conversational evidence does not ensure that an agent can infer and apply these mappings. We introduce BetweenUsBench, a benchmark for this capability, which we call dyad-specific memory. Its two dimensions are idiosyncratic reference, interpreting private codenames, in-jokes, and metaphors, and interactional rapport, following response expectations learned from prior interactions. Each memory system ingests a complete multi-session history and answers a subsequent query. An LLM judge assesses the reply against the history-supported mapping. History controls, a rapport-specific profile baseline, and supplementary diagnostics examine specificity, preservation, retrieval, and utilization. Across three memory backbones with a fixed reader, the best memory-method pass rates range from 20.42% to 25.42% for private reference and from 24.17% to 28.75% for rapport. Failures persist even when sufficient evidence reaches the reader. BetweenUsBench makes these limitations measurable, connecting response quality to how remembered experience is recovered and used.

Primary area:
datasets and benchmarks
Keywords:
long-term conversational memory, dyad-specific memory, personalized conversational agents, memory-augmented LLMs, conversational AI, dialogue benchmarks, memory retrieval and utilization, interactional rapport

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_7QHKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BetweenUsBench, a benchmark designed to evaluate dyad-specific memory in long-term conversational agents. The benchmark assesses 2 dimensions: idiosyncratic reference, which measures the ability to interpret user-specific expressions and meanings, and interactional rapport, which evaluates whether agents can respond according to preferences implicitly learned from past conversations. Using synthetically generated multi-session conversations, the authors evaluate 7 memory methods across three LLM backbones and analyze failures in memory preservation, retrieval, andutilization. The results reveal limitations in existing memory systems, even when relevant information is present in conversation histories, highlighting the need for better mechanisms to retrieve and apply user-specific conversational knowledge.

Critical strengths

Underexplored research direction. The paper addresses an important and relatively underexplored aspect of long-term conversation, particularly relevant to AI companions.

Systematic dataset construction. The synthetic data generation process supports systematic construction of challenging cases with controlled evidence and response requirements, offering potential scalability.

Diagnostic evaluation. The evaluation framework examines memory preservation, retrieval, and utilization, providing diagnostic insights into current memory systems.

Critical weaknesses

Insufficient validation of real-world validity. Although the authors conduct human audits and report high agreement between LLM judges and human annotators, it remains unclear how well the benchmark reflects real-world conversational dynamics and user-specific interaction patterns.

Limited analysis. The paper mainly reports overall performance, with limited analysis of which types of cases are particularly challenging. More fine-grained analyses, such as performance across different challenge categories (e.g., boundary avoidance, ritual enactment) and common failure patterns would help clarify what makes the benchmark difficult and what capabilities current models lack.

Limited exploration of prompting-based baselines. It remains unclear whether the observed failures stem from memory limitations or could be addressed through simpler prompting strategies. For example, explicitly prompting models to identify relevant evidence or reflect on user-specific patterns before responding could help clarify whether these challenges primarily arise from memory mechanisms or inference-time reasoning.

Questions for authors

What memory mechanisms would be most effective for the capabilities evaluated in BetweenUsBench? Have the authors considered whether optimizing memory systems for these capabilities might introduce trade-offs with traditional task-oriented memory performance, particularly under limited memory budgets?

Reviewer_2bQ5OpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BetweenUsBench, a benchmark for evaluating dyad-specific memory in long-term conversational agents, focusing on personalized understanding beyond factual recall. The benchmark targets two complementary capabilities: idiosyncratic reference, which requires interpreting user-specific expressions such as private codenames and metaphors, and interactional rapport, which requires adapting responses to context-dependent expectations inferred from prior interactions. BetweenUsBench consists of 600 synthetically constructed multi-session cases, including 480 held-out test cases, each containing a conversation history, a subsequent query, and evidence-grounded response criteria. The authors evaluate seven existing memory methods across three memory backbones using a fixed GPT-5 Mini reader and an LLM-based response judge. Additional controls and diagnostics investigate history dependence, evidence preservation, retrieval sufficiency, and downstream utilization. Experimental results show that even the best-performing memory method, T-Mem, achieves only 25.42% and 28.75% pass rates on the two dimensions under the GPT-5 Mini backbone, respectively, while providing gold evidence substantially improves performance. The paper highlights limitations of current memory systems in recovering and applying user-specific meanings and interaction expectations from long-term conversational histories.

Critical strengths
  • The paper identifies an important aspect of conversational memory that is not adequately captured by factual recall benchmarks. The distinction between idiosyncratic reference and interactional rapport provides a clear framework for evaluating whether memory systems can understand user-specific meanings and adapt their responses to individual interaction patterns.
  • The paper evaluates seven memory methods across three memory backbones, with a fixed response model and multiple well-designed control settings. The additional analyses of memory preservation, retrieval, and utilization provide useful insights into where existing systems fail. The human validation of the benchmark and LLM-based judgments further strengthens the experimental rigor.
Critical weaknesses

The paper violates the page limit with the main text of the initial submission extending onto page 10 (limit 9). Suggesting desk reject

  • The novelty of this paper may be limited. While the paper presents dyad-specific memory as a relatively underexplored evaluation target, several existing benchmarks already investigate closely related capabilities. For example, PrefEval evaluates the inference and application of implicit user preferences from multi-session conversations, ES-MemEval and MADial-Bench investigate personalized memory utilization in emotional support and conversational settings. The paper does not sufficiently discuss these closely related works or clearly establish how its evaluation capabilities differ from theirs.
  • The benchmark is constructed from predefined user-specific mappings and response requirements, with supporting evidence deliberately embedded in synthetic conversation histories, it may produce relatively clean and unambiguous mappings that do not adequately reflect the noisy, implicit, and potentially inconsistent signals found in real-world interactions. Moreover, the paper does not quantify how many cases genuinely require integrating evidence across multiple sessions, rather than recovering the target mapping from individual scenes. The near-perfect performance under gold-evidence conditions, particularly with stronger readers, further suggests that much of the observed difficulty may arise from locating relevant evidence rather than performing challenging dyad-specific inference.
Questions for authors
  • Could the authors clarify why fully synthetic conversations were chosen over real-world conversational histories? While synthetic data allows controlled construction and annotation, how do the authors ensure that the resulting cases preserve the ambiguity, inconsistency, and complex cross-session inference requirements of real user interactions?

Please also see the Weaknesses section for questions and concerns.

Reviewer_AHtROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The submission violates the 9-page limit for the main text., and we suggest a desk reject.

Critical strengths

This paper studies an interesting problem, and the presentation is generally easy to follow.

Critical weaknesses

This paper exceeds the page limit (the conclusion section is at page 10), and we suggest a desk reject. It seems not ready for review and not ready for publication.

Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.