- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PactBench, a benchmark for evaluating whether long-term conversational agents can form and use partner-specific conventions across sessions. The benchmark covers four convention-formation mechanisms and evaluates compression, acceptance, and production through session-isolated probes. The paper also proposes Pact-Mem, a memory module that jointly stores form–meaning mappings, consolidates evidence across sessions, and retrieves conventions using form-free situational cues. Experiments with four backbone LLMs and six existing memory systems show that current memory approaches struggle with convention production, while Pact-Mem improves production performance in several settings.
Critical strengths
The paper addresses an important gap between recalling explicitly stated facts and learning partner-specific conventions through repeated interaction.
PactBench provides a useful task decomposition and a diagnostic evaluation protocol that separates compression, acceptance, and production.
The Ref-Traj condition helps distinguish memory retrieval limitations from response-generation limitations.
The experimental study covers multiple backbone models and memory systems and includes retrieval diagnostics and long-horizon retention experiments.
Pact-Mem is a practical and interpretable design that demonstrates promising improvements in convention production on the proposed benchmark.
Critical weaknesses
Limited methodological novelty. Pact-Mem combines joint form–meaning storage, cross-session consolidation, and situational dense retrieval. Although these components are sensible, the paper does not sufficiently establish that the proposed combination constitutes a substantial methodological advance over existing structured-memory and retrieval approaches.
Insufficient validation of the synthetic benchmark. All conventions and formation trajectories are synthetically constructed. The paper does not demonstrate that the four mechanisms and fixed six-session trajectories adequately represent naturally occurring conventions in long-term human–agent interactions. This limits the generalizability of the findings.
Insufficient component-level ablations. The experiments do not adequately isolate the contributions of joint storage, consolidation, and form-free retrieval. Consequently, it remains unclear which components account for the reported improvements and whether a simpler structured-memory baseline could achieve similar results.
Limited statistical analysis. The paper primarily reports pass rates without a sufficiently developed analysis of confidence intervals and statistical significance. This is particularly concerning for low production rates and relatively small differences between methods.
Evaluation reliability concerns. The semantic evaluation relies on a single LLM judge and a human reference annotator. Although the reported overall agreement is encouraging, multiple independent annotators and a more comprehensive reliability analysis would provide stronger evidence that the conclusions are robust to evaluation bias.
Incomplete baseline and efficiency comparisons. Pact-Mem retrieves up to 15 entries, whereas the baselines generally retrieve up to five sessions. Although the paper imposes a character budget, it does not fully establish comparable token usage, processing costs, or retrieval opportunities. A simple structured-memory baseline and comparisons under matched compute and token budgets are needed.
Limited absolute performance and potentially broad conclusions. Pact-Mem improves production in several settings, but absolute production rates remain low, and Ref-Traj outperforms it in most production comparisons. The results support a targeted improvement on PactBench rather than a general solution to convention formation in long-term conversational agents.
Questions for authors
What evidence supports the ecological validity of the synthetic conventions and their predefined formation trajectories?
Can the authors provide component-level ablations to isolate the contributions of joint storage, cross-session consolidation, and form-free retrieval?
Do the reported improvements remain significant under paired statistical tests and matched token and compute budgets?
How robust are the conclusions under independent human evaluation with multiple annotators?
Can Pact-Mem generalize to independently collected human–agent conversations or convention benchmarks with different formation patterns?
2 comments
Summary of Rebuttal and Additional Experiments
Dear Reviewer,
Thank you for your valuable suggestions. We would like to briefly summarize the additional experiments and clarifications provided in our rebuttal for your convenience.
-
Methodological contribution (W1, W3, Q2): We clarified Pact-Mem's design rationale and added component ablations, together with comparisons against a simple form–meaning lexicon baseline.
-
Benchmark validity (W2, Q1): We clarified the theoretical motivation behind the proposed convention-formation mechanisms and documented human screening for benchmark naturalness and inferability.
-
Statistical significance and evaluation reliability (W4, Q3, W5, Q4): We added paired statistical tests and confidence intervals, independent evaluations by three human annotators, and consistency checks across LLM judges.
-
Retrieval settings and token usage (W6): We added retrieval top-k sensitivity experiments and retrieval-token usage comparisons on two backbones.
-
Generalization and scope (W7, Q5): We added factual-recall and personalization evaluations on LoCoMo and PersonaMem-32k, and clarified the distinction between limitations arising from memory access and those arising from the underlying backbone.
As the rebuttal period closes today, we would be grateful if you could take a moment to review these additional results and let us know if you have any remaining questions or concerns.
Thank you again for your time and constructive feedback.
Best regards,
The authors of Paper 32271
Follow-up on Our Rebuttal
Dear Reviewer,
As the discussion period draws to a close, we would be grateful if you could review our responses and additional experimental results. We would be happy to address any remaining questions or provide further clarification. Thank you again for your time and consideration.
Best regards,
The authors of Paper 32271