acceptodds
Under review as a conference paper at ICLR 2027

Can Agents Develop Private Shorthand with You? Benchmarking Dyadic Convention Formation in Long-Term Conversations

Abstract

Long-term conversational agents must do more than recall facts: they should acquire the partner-specific expressions that emerge through repeated interaction. Existing long-term memory benchmarks test information stated in dialogue rather than conventions that emerge across sessions, leaving open whether an agent can participate in their formation and later use them. We introduce PactBench, a benchmark of 200 dyadic conventions spanning inversion, anchoring, adoption and erosion. Each convention has user-compression and agent-compression histories that test whether the agent can compress a reference, accept its partner's shortened form and later produce the established form. We evaluate each required capability independently by withholding the agent turn at that session while preserving the reference formation trajectory. To prevent plausible but ungrounded replies from passing, conjunctive scoring combines a direct meaning question with capability-specific checks wherever a form has been established. Experiments with four backbone LLMs and six existing memory systems identify two failures: (i) LLM-based memory systems lose or fail to retrieve the form–meaning link, and (ii) even when given the complete preceding reference formation history, agents do not reliably compress or produce the form. We present Pact-Mem, which preserves convention-level form–meaning bindings across sessions. Across four backbone LLMs, Pact-Mem consistently improves convention production over existing memory systems by up to 31 percentage points. Pact-Mem provides a convention-aware memory module for preserving and using dyadic conventions in long-term conversations.

Primary area:
datasets and benchmarks
Keywords:
dyadic conventions, conversational agents, long-term memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QPGqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PactBench, a benchmark for evaluating whether long-term conversational agents can form and use partner-specific conventions across sessions. The benchmark covers four convention-formation mechanisms and evaluates compression, acceptance, and production through session-isolated probes. The paper also proposes Pact-Mem, a memory module that jointly stores form–meaning mappings, consolidates evidence across sessions, and retrieves conventions using form-free situational cues. Experiments with four backbone LLMs and six existing memory systems show that current memory approaches struggle with convention production, while Pact-Mem improves production performance in several settings.

Critical strengths

The paper addresses an important gap between recalling explicitly stated facts and learning partner-specific conventions through repeated interaction.

PactBench provides a useful task decomposition and a diagnostic evaluation protocol that separates compression, acceptance, and production.

The Ref-Traj condition helps distinguish memory retrieval limitations from response-generation limitations.

The experimental study covers multiple backbone models and memory systems and includes retrieval diagnostics and long-horizon retention experiments.

Pact-Mem is a practical and interpretable design that demonstrates promising improvements in convention production on the proposed benchmark.

Critical weaknesses

Limited methodological novelty. Pact-Mem combines joint form–meaning storage, cross-session consolidation, and situational dense retrieval. Although these components are sensible, the paper does not sufficiently establish that the proposed combination constitutes a substantial methodological advance over existing structured-memory and retrieval approaches.

Insufficient validation of the synthetic benchmark. All conventions and formation trajectories are synthetically constructed. The paper does not demonstrate that the four mechanisms and fixed six-session trajectories adequately represent naturally occurring conventions in long-term human–agent interactions. This limits the generalizability of the findings.

Insufficient component-level ablations. The experiments do not adequately isolate the contributions of joint storage, consolidation, and form-free retrieval. Consequently, it remains unclear which components account for the reported improvements and whether a simpler structured-memory baseline could achieve similar results.

Limited statistical analysis. The paper primarily reports pass rates without a sufficiently developed analysis of confidence intervals and statistical significance. This is particularly concerning for low production rates and relatively small differences between methods.

Evaluation reliability concerns. The semantic evaluation relies on a single LLM judge and a human reference annotator. Although the reported overall agreement is encouraging, multiple independent annotators and a more comprehensive reliability analysis would provide stronger evidence that the conclusions are robust to evaluation bias.

Incomplete baseline and efficiency comparisons. Pact-Mem retrieves up to 15 entries, whereas the baselines generally retrieve up to five sessions. Although the paper imposes a character budget, it does not fully establish comparable token usage, processing costs, or retrieval opportunities. A simple structured-memory baseline and comparisons under matched compute and token budgets are needed.

Limited absolute performance and potentially broad conclusions. Pact-Mem improves production in several settings, but absolute production rates remain low, and Ref-Traj outperforms it in most production comparisons. The results support a targeted improvement on PactBench rather than a general solution to convention formation in long-term conversational agents.

Questions for authors

What evidence supports the ecological validity of the synthetic conventions and their predefined formation trajectories?

Can the authors provide component-level ablations to isolate the contributions of joint storage, cross-session consolidation, and form-free retrieval?

Do the reported improvements remain significant under paired statistical tests and matched token and compute budgets?

How robust are the conclusions under independent human evaluation with multiple annotators?

Can Pact-Mem generalize to independently collected human–agent conversations or convention benchmarks with different formation patterns?

2 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

As the discussion period draws to a close, we would be grateful if you could review our responses and additional experimental results. We would be happy to address any remaining questions or provide further clarification. Thank you again for your time and consideration.

Best regards,

The authors of Paper 32271

AuthorsOpenReview ↗

Summary of Rebuttal and Additional Experiments

Dear Reviewer,

Thank you for your valuable suggestions. We would like to briefly summarize the additional experiments and clarifications provided in our rebuttal for your convenience.

  1. Methodological contribution (W1, W3, Q2): We clarified Pact-Mem's design rationale and added component ablations, together with comparisons against a simple form–meaning lexicon baseline.

  2. Benchmark validity (W2, Q1): We clarified the theoretical motivation behind the proposed convention-formation mechanisms and documented human screening for benchmark naturalness and inferability.

  3. Statistical significance and evaluation reliability (W4, Q3, W5, Q4): We added paired statistical tests and confidence intervals, independent evaluations by three human annotators, and consistency checks across LLM judges.

  4. Retrieval settings and token usage (W6): We added retrieval top-k sensitivity experiments and retrieval-token usage comparisons on two backbones.

  5. Generalization and scope (W7, Q5): We added factual-recall and personalization evaluations on LoCoMo and PersonaMem-32k, and clarified the distinction between limitations arising from memory access and those arising from the underlying backbone.

As the rebuttal period closes today, we would be grateful if you could take a moment to review these additional results and let us know if you have any remaining questions or concerns.

Thank you again for your time and constructive feedback.

Best regards,
The authors of Paper 32271

Reviewer_YxvBOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper presents PactBench - a new benchmark for long-term conventional agents that focuses on dyadic linguistic conventions emerging naturally between the participants of the conversation. In their experiments, the authors evaluated different backbone LLM - memory system combinations and identified two primary issues with existing memory systems regarding these dyadic conventions. As a solution, the authors proposed Pact-Mem - a memory system for conversational agents that preserves convention-level form-meaning bindings across sessions. The authors evaluate Pact-Mem against existing memory systems and show that their system performs significantly better in terms of convention production.

Critical strengths

A carefully prepared benchmark addressing a somewhat niche, yet highly interesting, topic related to conversational agents.

Critical weaknesses

None

Questions for authors
  • The authors do not state clearly from the beginning (abstract/introduction) what Pact-Mem actually is. It is implied that this is a memory system but it is not stated explicitly until section 5.1.
  • The authors did not state clearly in the main part of the paper what was the source of the 200 conventions in the benchmark. The procedure is given in Appendix B, but it is not referenced in this context in the main part of the paper.
  • Should the failure of a memory system to generate a private shorthand form truly be considered a failure of that system? Perhaps the use of a full - and thus more widely understood - form is actually the preferable option, as it carries a lower risk of ambiguity, especially in context of the usual application of conversational agents?
  • Did the authors assess the extent to which the LLM judge’s evaluation is repeatable?
3 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

As the discussion period draws to a close, we would be grateful if you could review our responses and additional experimental results. We would be happy to address any remaining questions or provide further clarification. Thank you again for your time and consideration.

Best regards,

The authors of Paper 32271

Thank you for addressing my questions, your answers are really convincing, I will increase my score.

Thank you for your valuable suggestions and for raising our score from 3 to 4. We sincerely appreciate your constructive feedback and will incorporate these changes into the final version of our paper.

Reviewer_tt4tOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the problem of dyadic conventions in long-term conversational agents, which are conventions unique to the two participants in a dyadic interaction. The authors point out that, over the course of long-term conversations, the two participants often develop expression patterns based on their interaction history. The meanings of these expressions may differ from their conventional public meanings, and their specific interpretations depend on the history of prior interactions between the two parties. To investigate this phenomenon, the authors introduce PactBench, a benchmark specifically designed to evaluate dyadic conventions, covering four mechanisms through which such conventions emerge. They evaluate several existing memory systems on their ability to understand and generate dyadic conventions, finding that all of them exhibit notable limitations. To address this issue, the authors further propose Pact-Mem, which significantly improves performance on the benchmark compared with existing memory systems, particularly in terms of the success rate of convention generation.

Critical strengths

1. This paper investigates dyadic conventions that emerge between two participants in long-term conversations. The research direction is novel and has practical value. The authors construct a dedicated benchmark for evaluating dyadic conventions, providing a specialized evaluation framework for studying how long-term conversational agents develop and use expressions whose meanings depend on interaction history.

2. Evaluations on PactBench reveal that existing memory systems still have limitations in understanding and generating conventions. To address this issue, the authors propose Pact-Mem, which achieves significant improvements over existing memory systems on PactBench. This work also provides valuable insights for future research on memory in long-term conversational agents.

Critical weaknesses
  1. Although Pact-Mem achieves significant improvements on PactBench, the authors evaluate it only on this benchmark. It remains unclear whether Pact-Mem generalizes to other benchmarks or whether it compromises performance on general-purpose memory evaluation tasks. Therefore, it is uncertain whether the advantages of Pact-Mem can be extended to other long-term conversational memory tasks.

  2. When evaluating existing baselines, the authors fix the retrievers to return the top k=5k = 5 sessions. It is unclear whether this setting provides a fair comparison. In particular, could the relatively poor performance of the baselines be attributed to the insufficient number of retrieved sessions rather than to the inherent limitations of the memory systems themselves?

Questions for authors
  1. Provide evaluation results for Pact-Mem on additional benchmarks to assess its generalizability beyond PactBench.

  2. Evaluate the baselines using different values of kk to investigate whether the choice of the number of retrieved sessions affects their performance.

  3. Pact-Mem appears to lack ablation studies of its individual components. It remains unclear whether each component contributes to the observed performance improvements. I recommend conducting ablation experiments on the three components: Joint form–meaning storage, Cross-session evidence consolidation, and Form-free situational retrieval.

3 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

As the discussion period draws to a close, we would be grateful if you could review our responses and additional experimental results. We would be happy to address any remaining questions or provide further clarification. Thank you again for your time and consideration.

Best regards, The authors of Paper 32271

I appreciate the authors' substantial efforts in addressing my concerns. Therefore, I will increase my score.

Thank you for your valuable suggestions and for raising our score from 2 to 3. We sincerely appreciate your constructive feedback and will incorporate these changes into the final version of our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Rebuttal for the AC's Reference

Dear Area Chair,

Thank you for your time and effort in handling our submission. As the rebuttal period approaches its end, we have already addressed all reviewers’ questions in our rebuttal and received positive ratings of 4 and 3 from reviewers who have responded. For your convenience, we summarize the additional experiments and clarifications below.

  1. Contribution and motivation (Reviewer YxvB, Q1/Q3; Reviewer QPGq, W1): We clarified the paper’s contribution, introduced Pact-Mem explicitly in the abstract and introduction, and explained the motivation for evaluating shorthand reuse.

  2. Benchmark validity (Reviewer YxvB, Q2; Reviewer QPGq, W2/Q1): We clarified the synthetic source of the conventions, the theoretical basis of the convention-formation mechanisms, and the human screening procedure for naturalness and inferability.

  3. Component ablations and structured baseline (Reviewer tt4t, Q3; Reviewer QPGq, W3/Q2): We added component ablations and comparisons against a simple form–meaning lexicon baseline.

  4. Statistical significance and evaluation reliability (Reviewer YxvB, Q4; Reviewer QPGq, W4–W5/Q3–Q4): We added paired statistical tests and confidence intervals, independent evaluations by three human annotators, and consistency checks across LLM judges.

  5. Retrieval settings and token usage (Reviewer tt4t, W2/Q2; Reviewer QPGq, W6): We evaluated the baselines at different retrieval top-k values on two backbones and reported retrieved-text token counts.

  6. Generalization and scope (Reviewer tt4t, W1/Q1; Reviewer QPGq, W7/Q5): We added factual-recall and personalization evaluations on LoCoMo and PersonaMem-32k with task-specific memory-writing instructions, and clarified the distinction between memory-access and backbone limitations.

We have not yet received a follow-up response from Reviewer QPGq, and the other two reviewers, tt4t and YxvB, assigned scores of 3 (Weak acceptance) and 4 (Clear acceptance). We hope these additional results and clarifications assist your assessment. Thank you again for your time, careful consideration, and efforts in coordinating the review process.

Best regards,
The authors of Paper 32271

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.