acceptodds
Under review as a conference paper at ICLR 2027

SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue

14 views

Abstract

Long-term conversational memory in multi-party settings requires more than retrieving relevant content from long-term conversations: it must distinguish who said what, whom each statement concerns, how individuals perceive one another, what information is shared by the group, and how states change over time. Recent studies on multi-party dialogue benchmarks show that existing general-purpose LLM memory systems tend to lose person and group relations or struggle to integrate clues distributed across members, groups, and time. Together, these issues reveal two core bottlenecks: message attribution and relational understanding in multi-party dialogue, and state reconstruction from interleaved histories. To address both, we propose : its dual-track memory stores speaker-labeled verbatim messages and derived states organized into person-level and group-level views, then combines evidence from both tracks by entity, event, and time at query time. To reduce attribution and update errors during structured memory construction while enabling local deployment, we train Writer-R1 with SpeakerLevenshtein and speaker-conditioned GRPO. On GroupMemBench, SocialMemBench, and EverMemBench, SpeakerMem-R1 achieves binary accuracies of 47.9%, 69.2%, and 61.9%, respectively. The scores are 3.3, 12.4, and 9.4 percentage points over the best results of mainstream frameworks evaluated on each benchmark, respectively. On the publicly reported EverMemBench leaderboard from EverMind-AI, SpeakerMem-R1 achieves 62.33%, the best reported result among the latest state-of-the-art frameworks. It also achieves 70.85% on all 1,986 LoCoMo questions, which we use as a two-person long-term conversation boundary test. In a controlled evaluation of 305 questions, RL raises the SFT Writer's mean accuracy from 57.38% to 68.20% under a frozen query/answer pipeline. We report both binary accuracy and token-F1, and ablations show that the verbatim and structured tracks, as well as person-level and group-level views, are complementary under the standardized evaluation interface.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-party dialogue, long-term conversational memory, dual-track memory, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_hx65OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method for multi-party dialogue analysis by improving the general-purpose LLM memory systems with speaker-centered dual-track memory. Experiments on serveral benchmarks have been conducted to evaluate the effectiveness of the proposed method.

Critical strengths

The characteristics of multi-party conversations are consided to deisgn the proposed method and experimental results demonstrated its superiority over general-purpose LLM memoy methods.

Critical weaknesses
  1. This paper focuses more on the conversational memory of LLMs rather than the task of multi-partiy dialogue understanding in Section 1 and 2. There are very few introductions on the development and recent progresses of multi-party dialogue undersanding in Section 1, and Section 2 only mentions the recent benchmarks for multi-party memory.
  2. The baseline methods used in the experiments are also general-purpose LLM long-term memory/RAG methods. The experimental results could be more convincing if some methods designed specifically for the task of multi-party conversation could be included.
Questions for authors
  1. In Table 4, the performance of the proposed method is not competitive on the LoCoMo dataset. Some discussions are expected in the main text of the paper.
  2. The authors usually put the citations of several methods together. It is difficult for readers to discriminate the reference for each method.
1 comment
AuthorsOpenReview ↗

Multi-Party Memory Context, Specialist Baselines, and LoCoMo Analysis

Thank you for the suggestions on multi-party research context, specialist baselines, LoCoMo, and citation clarity. We have revised all four areas; references below use the updated manuscript.

Weakness 1. Multi-party dialogue understanding.

SpeakerMem-R1 is designed for long-term multi-party conversational memory. Its central problem is preserving who said what, whom a statement concerns, and how individual and shared states evolve across interleaved conversations. The source/owner distinction and person/group views directly serve these requirements. Accordingly, GroupMemBench, SocialMemBench, and EverMemBench form our main evaluation.

Sections 1-2 now connect prior multi-party understanding to persistent conversational memory: DialogueGCN for speaker dependencies, Molweni for discourse structure, DialogRE for cross-utterance relations, QuISG/LIMN for dialogue QA, and FANToM for knowledge/belief reasoning. We also cover EMMA's partner-specific memory, the cross-session memory model accompanying M3C, and DialSim/LongDialQA. This positions our contribution as preserving attribution and evolving person/group states through long-term writing and retrieval, building on multi-party understanding.

Weakness 2. Multi-party-specific baselines.

Table 2 now includes Subject-Mem and SMG, which index facts by subject and represent social norms/relations, respectively. All entries follow the same Subject-Mem evaluation configuration; the panels identify original-report and our evaluation sources.

MethodMeanQMeanN
Subject-Mem0.3210.318
SMG0.2130.226
A-MEM0.6080.610
SpeakerMem-R1, no ASK0.7130.691
SpeakerMem-R1, ASK0.7100.693

Table 25 further controls participant-wise organization and the input-token budget using raw messages in place of derived states. Full no-ASK gains 9.12/2.95/7.50 percentage points on SocialMem/GroupMem/EverMem, supporting the structural contribution.

Question 1. LoCoMo performance.

LoCoMo is a two-person boundary test of a system developed for multi-party memory. It examines how well the same architecture works when the conversation has only two participants. Our main evaluation targets the attribution and state-reconstruction challenges of multi-party dialogue; LoCoMo extends the evaluation to a simpler participant structure. Section 5.6 and Appendix H now explain this role and discuss the results.

The system remains competitive with representative advanced memory methods in this setting. All-question accuracy is 70.85% (1,407/1,986). For the non-adversarial comparison in Table 6, our accuracy is 67.34%, above the reported Mem0 (66.88%) and Zep (65.99%) results; MemOS (75.80%) and LightRAG (79.87%) perform better overall.

The category-level results further highlight its strengths: single-hop accuracy is 77.88%, above Mem0's 67.13% and Zep's 61.70%; temporal accuracy is 70.72%, second-highest among the six systems in Table 6, ahead of LightRAG's 60.75% and Mem0's 55.51%. Thus, the multi-party-oriented design retains strong factual recall and temporal reasoning in two-person conversations.

The remaining gap is concentrated in multi-hop and open-domain questions, with accuracies of 41.13% and 40.62% (Table 28). With only two participants, member/group separation and cross-person relations may offer less relative benefit, while combining facts across segments and drawing on external knowledge become more prominent needs. This motivates stronger cross-evidence reasoning and external-knowledge integration while preserving the architecture's multi-party strengths.

Question 2. Method-citation correspondence.

We revised Sections 1-2 so each named method is followed immediately by its own citation. Subject-Mem and SMG are attributed to SocialMemBench. We also checked the new references against their original papers and synchronized both language versions.

We hope the revised context, specialist comparisons, and LoCoMo analysis address your concerns. We would appreciate your reassessment and are happy to clarify any remaining issue.

Reviewer_Ltm3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a speaker-centered dual-track memory system for multi-party dialogue that addresses message attribution and state reconstruction. System 1 stores speaker-labeled verbatim messages, while System 2 organizes provenance-linked derived states into person-level and group-level views. The authors also train a Writer-R1 using SpeakerLevenshtein and speaker-conditioned GRPO, keeping the query and answer modules frozen. The system is evaluated on GroupMemBench, SocialMemBench, and EverMemBench against retrieval and memory baselines, with LoCoMo used as a two-person long-term conversation boundary test.

Critical strengths
  1. A well-motivated design for multi-party memory. SpeakerMem-R1 combines verbatim messages with person-level and group-level derived states, preserving source-owner distinctions and historical updates.

  2. Near large-model performance through SFT and RL. The authors train a locally deployable Writer through SFT followed by speaker-conditioned GRPO. Experimental results support the feasibility of training a small Writer to retain much of a large-model Writer’ s downstream QA performance.

Critical weaknesses
  1. The paper mainly compares SPEAKERMEM-R1 with Mem0 and A-Mem, leaving out many recent advanced memory systems and retrieval-augmented generation (RAG) baselines. Broader comparisons are needed to establish its effectiveness and superiority over existing approaches.

  2. While overall QA results and component ablations are encouraging, recovering each participant’s stance remains challenging. On SocialMemBench' s per-person stance questions, accuracy is only 9.1% without ASK, below the embedding baseline’s 13.6%. This leaves uncertainty about how reliably the proposed memory structure preserves and reconstructs individual participants' states, a central motivation of the paper.

  3. The paper introduces Speaker Levenshtein and speaker-conditioned GRPO to improve memory construction, but does not evaluate whether applying similar R1-based training to existing memory frameworks can achieve comparable improvements. Such comparisons are important to isolate the contributions of the proposed training strategy from those of the dual-track memory architecture.

  4. Although the paper emphasizes state reconstruction and information updates over time, it lacks dedicated comparisons on temporal reasoning, particularly the ability to handle outdated information, conflicting statements, and evolving personal or group states. More targeted evaluations are needed to substantiate the claimed advantages in long-term memory management.

Questions for authors
  1. In Appendix I.6, Rule 5 instructs the answerer to interpret (none) as evidence that a participant never performed an action, while Rule 6 explicitly states that the same marker does not imply that the participant did nothing. These instructions appear to contradict each other. Since empty rows may result from incomplete memory construction or retrieval, how does the system distinguish genuine non-occurrence from missing evidence, and which rule takes precedence?

  2. Are the gains maintained under matched evidence budgets? BM25 and Embed retrieve only ten messages, whereas SpeakerMem combines System 1's top-10 messages with additional System 2 records retrieved per owner and source. This makes it difficult to separate the benefits of the proposed memory structure from those of additional evidence. Could the authors clarify how a fair comparison is ensured given these differences in retrieval budgets?

  3. The SFT/RL comparison does not isolate the contributions of SpeakerLevenshtein, the update-chain reward, and terminal QA gain. Could the authors provide reward-component ablations to identify which components drive the improvement?

  4. Since the controlled Writer evaluation is limited to SocialMemBench, could the authors provide results on additional datasets to assess whether Writer-R1's performance gains generalize to other dialogue settings?

1 comment
AuthorsOpenReview ↗

Broader Comparisons, Budget Controls, and Writer Ablations

Thank you for the thoughtful review. We address each concern below; table numbers refer to the revised manuscript.

Weakness 1. Broader comparisons. Table 1 evaluates BM25, Embed, Mem0, A-MEM, and HippoRAG on all three complete benchmarks, now with top-15 baseline retrieval. Table 2 adds Subject-Mem and SMG under the same Subject-Mem evaluation configuration; its panels distinguish result sources. Table 5 compares MemoBase, Zep, MemOS, RippleMem, and EverOS under the public GPT-4.1-mini/Gemini-3-Flash configuration. Our 62.33% overall accuracy is highest in that comparison, ahead of EverOS (60.08%) and RippleMem (54.75%). This combines full-benchmark testing of representative systems with broader comparisons on individual benchmarks.

Weakness 2. Per-person stances. Q3 requires all participants' stances and reasons to be recovered together: even Full context reaches only 18.2% accuracy in the category comparison. Our no-ASK accuracy is 9.1%, versus Embed's 13.6%, but token-F1 is 26.5%, versus BM25/Embed's 12.1%/10.0% and A-MEM/HippoRAG's 24.1%/23.4% (Tables 19, 21). This shows better partial content recovery. Raw-only reaches 68.18% Q3 accuracy (Table 25), identifying detailed stance preservation as an area for improvement; full memory performs better on attribution Q4 and cross-person beliefs Q5.

Weakness 3. R1 training in other frameworks. Our main contribution is speaker-centered dual-track memory, with targeted training of a locally deployable 3B Writer. Figure 4 tests the architecture; Table 4 holds the architecture and query/answer pipeline fixed to test training objectives. The new content-only RL control isolates the added value of speaker-specific scoring (Question 3 below). Transferring RL to other frameworks requires adapting their action spaces and supervision; we leave that extension to future work.

Weakness 4. Temporal reasoning and updates. Section 5.3 now brings the existing temporal results into the main discussion. The category comparison shows that GM temporal/update accuracy is 50.6%/33.6%, versus BM25's 41.4%/23.4% (Table 18). SM temporal evolution reaches 59.9%, versus A-MEM/HippoRAG's 45.0%/47.7% (Table 19). Public EverMem Temp accuracy is 35.00%, versus EverOS's 20.33%; EverOS remains stronger on Update/Multi (Table 5).

Question 1. Rules 5 and 6. The rules are complementary. Rule 5 applies specifically to "who never did X / who does not follow the norm": within an informative person-by-person comparison, an empty cell signals no derived record of X for that person. A nonempty cell about another matter alone is insufficient to draw this inference. Rule 6 adds raw evidence: if Alice's row is empty but an excerpt shows Alice did X, that fact should determine the answer. If every cell is empty, Rule 5 already directs the answerer to raw excerpts; Rule 6 permits "insufficient memory" only when both sources are silent. Negative answers are scoped to the supplied records. Appendix J retains the original prompt.

Question 2. Evidence budgets. SpeakerMem uses 10 S1 messages plus S2 records allocated by target participant. To improve fairness, we raised all fixed-count baselines from top-10 to top-15 and reran retrieval and QA (Table 1), retaining their native candidate-recall mechanisms. We also added a direct token-budget control: Raw-only retains S1, target-row queries, organization, and answering instructions, replacing derived states with raw messages packed within the full no-ASK input budget.

Acc. (%)Raw-onlyFull no-ASKGain (pp)
SocialMem60.0469.16+9.12
GroupMem44.0346.98+2.95
EverMem53.0060.50+7.50

These count-based results (Table 25; Appendix G.3) show gains beyond per-person raw-evidence allocation.

Questions 3 and 4. Reward ablations and Writer transfer. Table 4 reports accuracy (%). RL runs share the SFT initialization and training budget; query/answer modules are frozen throughout.

ObjectiveSM eval10 (305)GM Finance (214)EM topic 01 (488)
SFT57.3843.9353.28
Content-only RL62.3048.1356.76
RL without terminal QA62.9546.7355.94
Full RL68.2052.8061.07

Content-only jointly removes owner buckets, source/owner/layer gates, and macro/worst-owner aggregation from state scoring. Restoring them adds 5.90/4.67/4.30 pp. Removing terminal QA alone costs 5.25/6.07/5.12 pp. Update-chain rewards are retained; their standalone effect remains untested. The same Writers transfer without retraining to the complete Finance/topic 01 units, gaining 8.88/7.79 pp over SFT. Definitions and category counts are in Appendices E and G.4.

We hope these results address your main concerns and would appreciate your reassessment. We are happy to clarify any remaining issue.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Controlled Evidence Organization and Speaker-Specific Rewards

Thank you for identifying the two controls needed to clarify the architecture and training contributions. We have added both; table numbers below refer to the revised manuscript.

Weakness 1 / Question 1. Derived memory with target organization and evidence budget controlled.

SpeakerMem uses 10 S1 messages plus S2 records allocated by target participant. To improve fairness, we raised all fixed-count baselines from top-10 to top-15 and reran retrieval and QA (Table 1). We further control participant-wise organization and input tokens with the following experiment.

We added Raw-only + S2 organization. It retains S1, PROJECT's issue/rows/tense and ALL/GROUP organization, within-row MiniLM cosine ranking, and answering instructions. S2-style rows are filled with raw messages spoken by or literally mentioning the target participant; GROUP rows use the same unit's history. Selection uses raw text, keeping the original speaker labels, with no derived records or owner/provenance labels. Rows retain up to 40 candidates and the two-item/source allocation, followed by round-robin packing and deduplication against S1.

The answer-input token budget includes names, timestamps, and formatting, and follows the full no-ASK system. It is read from saved traces or reconstructed on the frozen store with the current projection; packing tolerance is 1% or eight tokens (Appendix G.3).

Acc. (%)Raw-onlyFull no-ASKGain (pp)
SocialMem (1,031)60.0469.16+9.12
GroupMem (745)44.0346.98+2.95
EverMem (2,400)53.0060.50+7.50

Full memory answers 94/22/180 more questions, respectively. These count-based results in Table 25 support the value of derived states beyond participant-wise raw retrieval and input length. Table 25 shows where the difference arises: full memory improves SM attribution Q4 from 61.25% to 90.00% and cross-person beliefs Q5 from 54.35% to 78.26%; Raw-only performs better on per-person stances and norm exceptions. Section 5.3 discusses this variation.

Weakness 2 / Question 2. Speaker-specific rewards, attribution, and omissions.

We added content-only RL, jointly removing source/owner/layer gates, owner buckets, and macro/worst-owner aggregation from state scoring. Content similarity, threshold, one-to-one alignment, record-count normalization, validity, update-chain, and terminal-QA rewards are retained. A second ablation removes terminal QA alone, keeping other coefficients fixed. All RL variants use the same Writer architecture, SFT initialization, training budget, and frozen query/answer pipeline.

Acc. (%)SM eval10 (305)GM Finance (214)EM topic 01 (488)
SFT57.3843.9353.28
Content-only RL62.3048.1356.76
RL without terminal QA62.9546.7355.94
Full RL68.2052.8061.07

With speaker scoring removed, gains over SFT remain 4.92/4.21/3.48 pp. Restoring it adds a further 5.90/4.67/4.30 pp, or 18/10/21 correct answers (Table 4). Both local state scoring and terminal QA contribute to the full improvement.

For attribution-related QA, restoring speaker scoring raises SM Q4 from 14/21 to 18/21 and Q5 from 18/25 to 23/25. Table 26 also reports SFT's Q4 of 19/21 and Q3 of 0/5 for all objectives. The requested direct fact-to-owner error and quiet-owner omission rates have not been measured; the present ablation tests the joint scoring design through QA, with gating and worst-owner removed together. Section 5.4 and Appendices E/G.4 give the definitions and full category results.

We hope these controls address your concerns about retrieval organization and speaker-specific scoring. We would appreciate your reassessment in light of these results and welcome any remaining questions.

AuthorsOpenReview ↗

Rebuttal Summary: New Evidence and Manuscript Revisions

We thank the Area Chair and all three reviewers for their careful assessment and constructive feedback. We have provided point-by-point responses under each review. Here we summarize the additional experiments, analyses, and revisions; all figure and table numbers refer to the revised manuscript.

1. Separating derived-memory gains from retrieval organization and budget. SpeakerMem uses S1 top-10 plus S2 records allocated by target participant. To improve fairness, we increased all fixed-count baselines to top-15 and reran retrieval and QA on all three full benchmarks (Table 1). We also added Raw-only + S2 organization: it retains S1, participant-wise query organization, within-row ranking, and answering instructions, but replaces derived records with raw messages packed within the full no-ASK system's input-token budget.

BenchmarkRaw-only Acc. (%)Full no-ASK Acc. (%)Gain (pp)
SocialMemBench60.0469.16+9.12
GroupMemBench44.0346.98+2.95
EverMemBench53.0060.50+7.50

These count-based gains correspond to 94, 22, and 180 additional correct answers. Figure 4 and Tables 3 and 25 report the architecture comparisons; Appendix G.3 details budget construction and packing. The results support the value of derived states beyond organizing raw evidence by participant.

2. Testing speaker-specific rewards and Writer transfer. We added two RL ablations: content-only state matching, which jointly removes owner buckets, source/owner/layer gates, and macro/worst-owner aggregation; and removal of terminal QA feedback. The RL variants share the Writer architecture, SFT initialization, training budget, and frozen query/answer pipeline (Table 4).

Full RL reaches 68.20% on SocialMem eval10 (305 questions), compared with 57.38% for SFT, 62.30% for content-only RL, and 62.95% without terminal QA. Without retraining, the same Writers are evaluated on GroupMem Finance (214 questions) and EverMem topic 01 (488 questions): full RL improves over SFT from 43.93% to 52.80% and from 53.28% to 61.07%, respectively. Restoring speaker-specific scoring adds 4.30-5.90 percentage points across these three sets. These controls distinguish the architecture's contribution from the benefits of targeted training of a locally deployable 3B Writer.

3. Broadening comparisons with general and multi-party memory methods. The revised evaluation includes HippoRAG alongside BM25, Embed, Mem0, and A-MEM across the three complete benchmarks. Table 2 adds Subject-Mem and SMG under the common Subject-Mem configuration, distinguishing published results from our evaluations. SpeakerMem's best MeanQ/MeanN are 0.713/0.693, compared with Subject-Mem's 0.321/0.318 and A-MEM's 0.608/0.610. Table 5 presents the broader public EverMemBench comparison: under its GPT-4.1-mini/Gemini-3-Flash configuration, our 62.33% overall accuracy is highest among the listed systems, ahead of EverOS (60.08%) and RippleMem (54.75%).

4. Clarifying state recovery, per-person stances, and the two-person boundary. Section 5.3 now discusses temporal and update categories explicitly. GroupMem temporal/update accuracy reaches 50.6%/33.6%, versus BM25's 41.4%/23.4%; SocialMem temporal evolution reaches 59.9%, versus A-MEM's 45.0%. On Q3, no-ASK accuracy is 9.1%, but token-F1 is 26.5%, versus Embed's 10.0%. We analyze raw evidence's stronger stance recovery and derived memory's stronger attribution/cross-person-belief QA (Tables 18-21 and 25).

LoCoMo is explicitly positioned as a two-person boundary test of a multi-party memory system (Section 5.6; Appendix H). SpeakerMem achieves 70.85% on all 1,986 questions; its 67.34% non-adversarial accuracy exceeds the reported Mem0/Zep results, while MemOS/LightRAG remain stronger overall. We discuss strong single-hop/temporal recall and the remaining multi-hop/open-domain challenges.

5. Improving clarity and presentation. We expanded multi-party dialogue understanding and memory coverage in Sections 1-2, attached citations directly to individual methods, and clarified metric definitions and links to appendix formulas. Our response explains how answer-prompt Rules 5 and 6 complement each other through question-specific empty-row interpretation and raw-evidence support. We also reorganized the figures, tables, and implementation details while keeping the main text within 10 pages.

We hope this evidence and the revisions address the central concerns about comparison breadth, retrieval fairness, speaker-specific training, and task scope. We respectfully invite the reviewers and Area Chair to reassess our work in light of these results, and would welcome clarification of any substantive concerns that remain.

Thanks for your comment! We have actually compared our method with many state-of-the-art advanced memory systems and RAG approaches, including MemoBase, Zep, MemOS(2025-07), RippleMem(2026-08) and EverOS(2026-01), on EverMemBench (Table 5). Our overall accuracy of 62.33% is the highest among these baselines, outperforming EverOS (60.08%) and RippleMem (54.75%). These cover most of the latest mainstream advanced memory systems and RAG methods. Besides, we also supplemented additional tests of multi-agent memory systems Subject-Mem(2026-05) and SMG (2026-05)on SocialMemBench (Table 2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.