acceptodds
Under review as a conference paper at ICLR 2027

Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations

Abstract

Long-term memory enables agents to accumulate information and reason across sessions, yet existing research primarily focuses on dyadic text or image-text conversations, leaving long-term memory for multi-party spoken conversations underexplored. This setting requires preserving conversational content, identifying participants across sessions, and retaining who speaks to whom. To this end, we propose VoxPolyMem, an interaction-aware multimodal memory framework combining incremental speaker identification with a memory hierarchy comprising interaction memory, fact memory, and participant profiles. We formulate retrieval as sequential decision-making, where an agent rewrites queries and selects retrieval tools and memory layers based on accumulated evidence to address information gaps. We further introduce Evidence-Gain GRPO (EG-GRPO), which uses round-wise credit assignment to encourage complementary evidence acquisition. We also construct VoxPolyBench to evaluate memory evolution, personalized answering, memory retrieval and reasoning, and interaction reasoning and attribution in multi-party spoken conversations. VoxPolyMem achieves an overall score of 85.0 on VoxPolyBench, surpassing the strongest evaluated baseline by 23.6 points. On Mem-Gallery and H2HMem-Multi, it scores 89.6 and 74.4, respectively, exceeding the strongest evaluated public memory baselines by over 8 points each. These results highlight its potential for persistent, personalized assistance in multi-party multimodal interactions. Code and datasets are available at https://voxpolymem.github.io/VoxPolyBench/demo/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Term Agent Memory, Multi-Party Spoken Conversations, Multimodal Memory, Agentic Retrieval, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rWy3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates long-term memory in multi-party, cross-session spoken dialogues and introduces VoxPolyMem, a system that integrates online speaker recognition, interaction dynamics, factual memory, and participant profiling, utilizing a retrieval agent to manage memory layer selection, tool usage, and query rewriting. The authors further propose EG-GRPO, a training method that uses the coverage and ranking of newly acquired supporting evidence per turn as rewards for the retrieval strategy. Additionally, the paper presents VoxPolyBench, a dataset comprising 18 scenarios, 176 sessions, 18.9 hours of synthetic speech, and 1,527 question-answer pairs. Experimental results demonstrate significant performance gains on this benchmark as well as on two public multimodal memory benchmarks, with ablation studies analyzing the roles of memory structure, iterative retrieval, and strategy training.

Critical strengths

The problem formulation is practically significant. Long-term, multi-party conversations require tracking both "who said what" and "to whom it was said"; simply storing facts can easily lead to confusion regarding information attribution. This paper explicitly models participants and interaction relationships—utilizing them for retrieval filtering—representing a promising direction for research.

The system's performance gains do not rely solely on the authors' custom benchmark. The method also demonstrates improvements on two public multimodal memory benchmarks, with ablation studies confirming the contributions of interaction relationships, memory hierarchy, and iterative retrieval. These results substantiate the framework's practical value.

Critical weaknesses

In Table 2, VoxPolyMem scores 85.0 on the self-constructed benchmark, yet the version without RL already achieves 84.0. The RL gains on the two public benchmarks are 3.3 and 2.3 points, respectively. Consequently, the substantial lead over external baselines was largely established prior to the introduction of EG-GRPO. While the authors explicitly report this fact, it implies that the overall system's superiority cannot serve as direct evidence of the new training algorithm's significance. Existing work like Mem-T (Yue et al., 2026) already integrates hierarchical memory, iterative retrieval, and MoT-GRPO, providing process supervision through tree-based sampling, node-level evidence rewards, and outcome backpropagation. Thus, the concept of "shifting from sparse terminal rewards to step-wise credit assignment" already has direct precedents. The specific novel contributions of this paper lie in deduplicating newly added evidence, incorporating ranking rewards, and comparing candidate actions within the same state.

Although the authors do compare their method against MoT-GRPO in Table 4, that comparison simultaneously alters both the reward composition and the credit assignment mechanism, making it impossible to determine which specific design is effective. The study lacks independent ablation studies comparing new versus cumulative evidence, coverage versus ranking rewards, and intra-state grouping versus other grouping methods; furthermore, it reports neither training variance across runs nor a standardized computational budget. The advantage over MoT-GRPO on the self-constructed benchmark is a mere 0.9 points, and the available evidence is insufficient to determine whether this difference is stable. A more fundamental issue is that the reward and Recall metrics measure source ID coverage rather than whether the retained content actually contains the evidence required to answer the question. Appendices B.3 and C.2 clarify that a single fact or profile record may contribute multiple source utterance IDs via refer_id, yet the answering model receives a compressed record rather than the full content of those source utterances. The fact that a record is linked to a supporting utterance does not guarantee that it retains the specific information from that utterance needed to answer the current question. The description of the train-test split is limited to the number of QA pairs. Table 5 divides VoxPolyBench into 200 training questions and 1,327 test questions but fails to specify whether they originate from mutually exclusive scenes, participants, or conversation histories. The same lack of detail applies to the data split for Mem-Gallery. If training and test questions share the same history, the results primarily demonstrate adaptation to new questions within a known memory bank, rather than supporting generalization to unfamiliar groups or novel interaction histories. The paper currently lacks sufficient information to distinguish between these scenarios. The speech setup is tightly controlled: participants use consistent reference voices across sessions, and the synthesis process incorporates re-synthesis and transcription consistency checks, resulting in a final Word Error Rate (WER) of 2.14%. While these design choices ensure data quality, they also minimize the acoustic variability typical of real-world speech scenarios. The IEMOCAP and AMI experiments in Appendix D.1 rely on known utterance boundaries and evaluate only speaker tracking; they do not examine how identity errors in actual recordings propagate to interaction relationships, fact attribution, participant profiling, and final answers. Accuracy metrics for the recognition module on real recordings cannot substitute for a comprehensive evaluation of the memory-based QA system using actual speech input.

AFA employs ECAPA-TDNN speaker recognition to route speech queries to the corresponding user's cross-session memory and dynamic profile. The current paper lacks a direct comparison that isolates the value of the newly introduced interaction structures. AFA focuses on individual user memories, whereas this paper addresses shared multi-party histories—scenarios that are not equivalent; this distinction necessitates a comparison to determine exactly how "identity routing plus individual memory" differs from "identity routing plus shared interaction relationships."

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer rWy3 (1/3): EG-GRPO contribution and stability

Response to the concerns about EG-GRPO's contribution, reward ablations, and training variability

Thank you for asking us to distinguish the contributions of memory organization and policy training. EG-GRPO improves supporting-evidence acquisition and organization within the existing memory system, increasing answer quality while reducing retrieval rounds. We added reward-design comparisons and three independent training seeds to examine the contribution of ranking and the repeatability of the training gains.

1. Methodological contribution: complementary evidence acquisition and ranking.

Mem-T has explored node-level evidence rewards, tree-based sampling, and outcome backpropagation. EG-GRPO focuses on supporting evidence still missing from the current context: after fusion, deduplication, and context selection, only newly acquired supporting evidence receives additional coverage reward. The ranking term encourages supporting evidence to appear earlier within the limited context.

For example, if evidence A is already available but B is missing, retrieving A again through another memory layer receives no additional coverage reward, whereas recovering B receives direct credit. This aligns training with the information gaps that subsequent retrieval rounds need to resolve.

2. Reward-design comparisons.

ConfigurationVPB ScoreVPB RecallVPB RoundsMG ScoreMG RecallMG RoundsH2H ScoreH2H Rounds
Terminal-coverage GRPO83.186.71.486.489.71.571.61.5
EG-GRPO w/o ranking84.886.91.488.193.21.4——
Full EG-GRPO85.287.51.289.693.61.374.51.2

VPB, MG, and H2H denote VoxPolyBench, Mem-Gallery, and H2HMem-Multi. Score and Recall are multiplied by 100. Rounds counts retrieval rounds per question, excluding answer generation. Terminal-coverage GRPO retains the paper's reference-run results.

Without ranking, EG-GRPO still exceeds terminal cumulative-coverage training by 1.7 and 1.7 score points on VoxPolyBench and Mem-Gallery, respectively, supporting the effectiveness of the round-wise training scheme.

With the remaining settings held fixed, adding ranking further improves scores by 0.4 and 1.5 points on VoxPolyBench and Mem-Gallery, while reducing average retrieval rounds by approximately 14.3% and 7.1%. Evidence recall also increases by 0.6 and 0.4 points on VoxPolyBench and Mem-Gallery. This directly demonstrates the additional value of ranking beyond coverage alone: evidence coverage and answer quality improve with fewer retrieval rounds.

Overall, Full EG-GRPO exceeds Terminal-coverage GRPO by 2.1, 3.2, and 2.9 score points, with approximately 14.3%, 13.3%, and 20.0% fewer retrieval rounds.

3. Repeatability across three training seeds.

MethodVoxPolyBenchMem-GalleryH2HMem-Multi
w/o RL84.086.372.1
MoT-GRPO84.0 ± 0.186.9 ± 0.272.4 ± 0.8
EG-GRPO85.2 ± 0.289.6 ± 0.274.5 ± 0.1

Trained methods report mean ± sample standard deviation across three seeds; w/o RL is a fixed reference configuration. EG-GRPO outperforms MoT-GRPO in every matched seed on all three benchmarks, with mean gains of 1.2, 2.7, and 2.1 points. On H2HMem-Multi, its standard deviation is 0.1, compared with 0.8 for MoT-GRPO. Relative to w/o RL, EG-GRPO also reduces mean retrieval rounds from 1.6, 1.7, and 1.4 to 1.2, 1.3, and 1.2.

These experiments support EG-GRPO through reward ablation and repeated training: ranking provides measurable additional benefits, and the complete method consistently improves answer quality across the evaluated seeds while using fewer retrieval rounds. We will incorporate these results into the revision and distinguish memory-system gains from the further improvements due to policy optimization. Score differences are computed before display rounding; round reductions use the reported mean round counts.

AuthorsOpenReview ↗

Response to Reviewer rWy3 (2/3): Evidence validity and history-disjoint evaluation

Thank you for raising the concerns about evidence validity and generalization. We clarify both below.

1. Source-evidence coverage and answer quality. The retrieval policy selects interaction, fact, or profile memory, and the answer model receives the actual records from the selected layers. The refer_id links map facts and profiles to their source utterances, providing a common evidence unit across memory granularities and preventing repeated credit for the same source across layers or rounds. “Expansion” in Appendix B.3 refers to this source-ID mapping for reward computation, not replacement of every retrieved record with the full source utterance.

Source coverage measures provenance overlap with annotated supporting evidence; it does not by itself guarantee that a record preserves every detail needed for the current question. Accordingly, inference-time sufficiency checks operate on the retrieved content and can trigger further retrieval, while QA scores independently assess generated answers rather than ID matches. Across three training seeds, EG-GRPO improves mean QA scores over MoT-GRPO by 1.2, 2.7, and 2.1 points on VoxPolyBench, Mem-Gallery, and H2HMem-Multi, respectively. The gains therefore extend to answer quality. We will explicitly define the metric as source-evidence coverage and distinguish it from content sufficiency.

2. Training and test conversation histories are disjoint. Policy training uses the sources listed below. Test questions come from different conversation histories, rather than a question-level split within a shared memory bank.

BenchmarkRL training sourcesTraining QA pairsTest-history separation
VoxPolyBenchG003, G010200Histories disjoint from the training scenarios
Mem-GalleryHome-Repair, Music-Dance, Real-Estate200Histories disjoint from the training topics
H2HMem-MultiNone0Evaluation only

These experiments thus evaluate policy transfer to conversation histories and memory banks unseen during training, rather than adaptation to new questions about known histories. We will add the scenario/topic identifiers and history-disjoint split protocol to the revision and reconcile the reported training counts accordingly.

AuthorsOpenReview ↗

Response to Reviewer rWy3 (3/3): Real speech and shared interaction memory

Thank you for raising the questions about real-speech evaluation and the value of shared interaction memory. We provide additional acoustic tests and baseline comparisons below.

1. Real recordings and overlapping speech. Beyond the IEMOCAP and AMI experiments in Appendix D.1, we evaluated AMI Meeting recordings with natural overlap and the same five VoxPolyBench cases under controlled 10% and 20% overlap. The table reports speaker-ID accuracy (%).

Cumulative configurationVoxPolyBench: 10% overlapVoxPolyBench: 20% overlapAMI: with overlap
Static voiceprint prototypes84.379.365.4
+ Online EMA updates95.492.285.3
+ Acoustic identity consolidation96.994.886.3

The complete module improves over static prototypes by 20.9 points on AMI. Increasing controlled overlap from 10% to 20% reduces its accuracy by 2.1 points, compared with 5.1 points for static prototypes. These results support online identity maintenance under overlapping speech and real recordings.

These experiments evaluate identity association. Measuring how identity errors propagate to fact attribution, profiles, and final QA additionally requires real recordings with jointly verified cross-session identities, interaction relations, fact updates, and question-specific supporting evidence. Existing resources have limited joint coverage of these annotations, whose construction requires substantial manual annotation and verification. We plan to incorporate a manually verified real-recording memory-QA subset to evaluate the complete pipeline.

2. AFA, IGMem, and the contribution of shared interaction relations. We added comparisons with identity-aware memory method AFA [1] and multimodal multi-party memory method IGMem [2], including an evaluation on independently developed EverMemBench. Scores are mean answer scores multiplied by 100; VoxPolyMem retains the main-paper reference runs on the original three benchmarks.

MethodVoxPolyBenchMem-GalleryH2HMem-MultiEverMemBench
AFA [1]37.647.539.944.3
IGMem [2]58.275.570.753.9
VoxPolyMem85.089.674.467.8

VoxPolyMem exceeds IGMem, the stronger additional baseline, by 26.8, 14.1, 3.7, and 13.9 points, respectively. The gains span multi-party conversations, dyadic multimodal conversations, and an independently constructed benchmark.

Identity routing establishes who is asking; shared interaction relations further help establish whom information concerns. For example, Alex tells Emma, “I remember you're allergic to peanuts, so shall we avoid dishes with peanuts today?” Alex is the speaker, but the allergy concerns Emma. VoxPolyMem uses these relations to construct facts and profiles, while allowing the retrieval policy to apply speaker or addressee conditions when useful.

To examine this contribution directly, we retain message content and speaker attribution while removing addressee-side links and the retrieval operations that depend on them. Persona and interaction-attribution scores decrease from 75.9 and 86.0 to 70.5 and 72.5, respectively; both configurations use w/o RL. The baseline comparisons assess the complete system, while this ablation tests interaction relations and their associated access mechanisms beyond speaker-only attribution. Together, they provide complementary evidence for shared interaction memory. We will incorporate these additional results and clarify their evaluation scope in the revision. Differences are computed before rounding.

[1] AFA: Identity-Aware Memory for Preventing Persona Confusion in Multi-User Dialogue. https://arxiv.org/abs/2604.25022

[2] IGMem: Interaction-Grounded Memory for Multimodal Multi-Party Conversations. https://openreview.net/forum?id=3lSas96zVV

Reviewer_ApvgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies long-term memory for multi-party spoken conversations across multiple sessions. The authors propose VoxPolyMem, which couples open-set incremental speaker identification with a three-layer memory hierarchy: a directed interaction graph capturing who speaks to whom, extracted factual propositions, and participant profiles. To retrieve evidence, the framework employs an agentic policy trained with Evidence-Gain GRPO, which provides step-wise rewards for newly acquired supporting evidence. The authors also construct VoxPolyBench, a synthetic multi-party spoken dialogue dataset comprising 18 scenarios and 1,527 QA pairs, demonstrating improvements over existing memory baselines.

Critical strengths

1 Explicit interaction structure: Modeling multi-party conversations as a directed graph linking speakers, messages, and addressees is a logical approach to resolving identity confusion and attribution in group interactions. 2 Step-wise reward design: Modifying GRPO to reward newly retrieved evidence rather than relying solely on terminal QA accuracy offers a more grounded credit assignment mechanism for multi-round retrieval.

Critical weaknesses

1 Disconnect between interactive motivation and practical pipeline latency: The paper frames the work around conversational systems like in-car or household assistants that demand responsive interaction. However, the system relies on an extensive chain: per-turn ASR, voiceprint extraction, sliding-window LLM graph extraction, and an 8B MLLM agent performing multi-round tool selection and query rewriting. This complexity entails considerable response latency and computational overhead, making real-time multi-party interaction difficult to achieve in practice. 2 Synthetic acoustic evaluation without noise or overlapping speech: VoxPolyBench consists entirely of clean TTS-synthesized speech based on GPT-4.1-generated scripts. Real multi-party environments frequently involve background noise, room reverberation, and overlapping speech. Evaluating speaker identification on isolated, pre-segmented clean tracks leaves it unproven whether the system is robust in natural acoustic settings, or whether the high scores are partly an artifact of the clean synthetic pipeline.

Questions for authors

1)What is the measured latency and computational/token cost for memory construction per turn and for answering a spoken query through the multi-round retrieval loop? 2 How does the incremental speaker tracking behave when applied to continuous audio with background noise or overlapping speech, rather than clean, pre-segmented single-speaker utterances? 3 To what extent are the empirical improvements driven by the synthetic dialogue distribution of GPT-4.1 and Qwen-TTS? Have the paper tested the retrieval and memory pipeline on real human conversational recordings?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Apvg

Thank you for recognizing the interaction structure and step-wise evidence rewards. We address your concerns point by point below.

W1 & Q1: Online update latency and retrieval cost

VoxPolyMem updates memory incrementally as messages arrive, including speaker identification and voiceprint maintenance. Our experiments follow the same chronological online protocol; construction does not wait until all sessions end.

Writing latency starts when a spoken utterance ends, or when a text message arrives. Interaction memory first records content, speaker identity, and timestamp. Profile and episodic memory then update in parallel. Episodic extraction jointly infers addressees and links them back to source utterances and participant nodes in interaction memory, without a separate addressee-identification LLM call.

ComponentMean latencyScope
Speaker identification and voiceprint update0.038 s/segmentExtraction, matching, gated EMA update, and speaker confirmation
Initial interaction-memory availability0.87 sIncludes streaming ASR and recording content, speaker, and timestamp
Profile and episodic memory availability3.291 sFrom utterance end to availability; parallel updates, including addressee links
Multi-round retrieval3.40 s/queryLocal 8B policy; 1.2 rounds on average; excludes answer generation and judging

New content is therefore searchable through interaction memory before all derived updates finish. Derived memories become available within a few seconds on average; queries do not reconstruct the full history. The writing-stage times describe different availability endpoints and are not additive.

On Mem-Gallery's AI-Robotics topic, retrieval averages 7,966 input and 231 output tokens per query. A separate writing-cost measurement on two 24-message prefixes used six extraction calls, totaling 5,311 input and 4,403 output tokens. Estimated API cost was USD 0.0092 total, or USD 0.00019/message, using CNY 7.3/USD. This prefix-specific estimate excludes local hardware cost.

W2 & Q2: Real recordings and overlapping speech

Appendix D.1 already evaluates identity association on real IEMOCAP and AMI recordings. We additionally evaluated AMI Meeting recordings with natural overlap and the same five VoxPolyBench cases with controlled 10% and 20% overlap. Speaker-ID Accuracy (%) is:

Cumulative configurationVoxPolyBench: 10% overlapVoxPolyBench: 20% overlapAMI Meeting: with overlap
Static voiceprint prototypes84.3479.2665.41
+ Online EMA updates95.4492.2485.34
+ Acoustic identity consolidation96.8594.7786.31

EMA improves over static prototypes by 11.10, 12.98, and 19.93 points, respectively; consolidation provides further gains. Increasing controlled overlap from 10% to 20% reduces static-prototype accuracy by 5.08 points, compared with 2.08 points for the full acoustic system.

This behavior is consistent with confidence-gated voiceprint maintenance: low-confidence observations do not directly update existing prototypes, while reliable observations continue updating identity memory. The system thus adapts online while limiting contamination from unreliable acoustic observations.

Q3: Synthetic distributions and real-audio memory evaluation

We examine generalization across dialogue distributions and acoustic conditions. On independently constructed Mem-Gallery, H2HMem-Multi, and EverMemBench, VoxPolyMem scores 89.60, 74.40, and 67.75, exceeding the additional IGMem baseline by 14.06, 3.74, and 13.86 points. Memory and retrieval gains therefore extend beyond our generated dialogue/question distribution. IEMOCAP and the new AMI overlap results provide complementary evidence for identity association in human recordings.

Public resources jointly covering natural multi-party audio, cross-session identity, evolving fact ownership, multi-hop and personalized QA, and verified supporting evidence remain limited. Building such an evaluation requires tracking participants and fact changes over long histories and manually verifying questions, answers, and evidence chains; automatic extraction alone cannot ensure annotation consistency.

VoxPolyBench's controlled construction enables these elements to be consistently annotated and checked, supporting systematic comparisons of identity association, interaction attribution, and long-term memory. We will extend it with a human-annotated and verified real-recording subset for long-term memory, multi-hop reasoning, and personalized QA.

Thank you for the detailed response, which has addressed most of my concerns. I will re-evaluate the paper's contributions and consider raising my score.

Also I would like to clarify how AI was used in preparing my review. I read and evaluated the paper myself and wrote the initial draft, including the strengths, limitations, and questions. I only used AI for translation, and it was not involved in any decision-making or forming of opinions. I also manually checked the translated text to make sure it matched my original thoughts.

AuthorsOpenReview ↗

Follow-up on the Revised Manuscript

Thank you for your thoughtful response and for considering our clarifications. We have updated the manuscript to incorporate the additional experiments and revisions discussed in our rebuttal.

As the discussion period is drawing to a close, please let us know if you have any further questions or if any points would benefit from clarification. We would be happy to discuss them.

Reviewer_MNkROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VoxPolyMem, a memory system for multi-party spoken conversations. The system combines speaker identification, a memory hierarchy containing interactions, facts, and participant profiles, and an agent that searches these memories over multiple rounds. The authors propose EG-GRPO to train the retrieval agent by rewarding newly acquired supporting evidence. They also introduce VoxPolyBench, a synthetic benchmark for memory and participant-specific reasoning across sessions, and evaluate the system on this benchmark and two existing image-text memory benchmarks.

Overall, I find the problem interesting and the reported results promising. However, the system's contribution beyond existing memory agents seems limited, and the specific benefits of the retrieval training method are not sufficiently established. I also have concerns about whether the benchmark's validation supports the additional capabilities it aims to evaluate. Because of these concerns, I am currently leaning towards rejection. Please see details below.

Critical strengths
  • [Benchmark novelty] I find VoxPolyBench's combination of recurring speaker identification and long-term group memory to be a meaningful extension on top of existing benchmarks. It connects voice-based identity across sessions with questions about changing facts, individual preferences, and who said what to whom. This makes it possible to evaluate not only whether an assistant remembers the right information, but whether it associates that information with the right participant over time. This joint setting to me is the most promising new contribution of the paper.

  • [Empirical results] The reported results in Table 2 are strong across VoxPolyBench, Mem-Gallery, and H2HMem-Multi. The version without RL already outperforms the compared methods on all three benchmarks, and Table 3 shows that the memory structure and multiple retrieval rounds contribute to performance. Using the same answer model also makes these comparisons more informative. The separate speaker-tracking experiments on IEMOCAP and AMI provide useful evidence beyond synthesized speech.

Critical weaknesses
  • [Memory system] I find the technical contribution of VoxPolyMem to be limited in relation to existing memory systems. AFA [1] already combines voice-based user identification with persistent profiles and user-specific memory. M3-Agent [2] already connects recurring identities to multimodal episodic and semantic memories and learns multi-round retrieval. M2A [3] also links higher-level memories to original messages and uses iterative retrieval across different memory representations. The explicit speaker-message-addressee relations and the corresponding filters seem to be the clearest additional design here. While this is a reasonable choice, it is not clear to me what technical challenges had to be solved to integrate these components, or what design insights emerge beyond retaining participant information during retrieval. Table 3 shows that removing interaction relations reduces performance, but this variant removes both the graph edges and relation-based filtering. It therefore does not distinguish the benefit of the proposed representation from simply keeping the same speaker and addressee information as metadata and using it to filter results.

  • [Retrieval training] I also find the methodological contribution of EG-GRPO limited. Mem-T [4] already uses step-level evidence rewards and branching retrieval trajectories, while LeTS [5] explicitly discourages retrieving information already found in earlier steps. EG-GRPO makes more specific changes: it rewards only new supporting evidence retained in the selected context, accounts for its rank, and compares candidate actions starting from the same search state. To me, these differences appear to be refinements of an existing training approach. Table 4 shows that the full recipe improves performance, yet does not distinguish which of these changes accounts for the gain. For example, it is unclear whether the ranking term adds anything beyond a coverage-only reward under the same training setup. It would be good to clarify what these additions contribute beyond the existing methods.

  • [Benchmark contribution] The benchmark VoxPolyBench also needs stronger validation. GroupMemBench [6] already evaluates shared group memory and questions whose answers depend on the asker, while H2HMem [7] covers multi-session, multi-party memory with text and images. PACHAT's Persona-Dialogue [8] also includes profile-grounded multi-party speech. The main distinction here is combining these capabilities with recurring voice-based identity across sessions. However, the evaluation relies on a small set of generated conversations with fixed synthetic voices, and the real-recording experiments test speaker tracking rather than the complete memory system. It is unclear how much additional difficulty this joint setting introduces or how well it reflects natural spoken conversations. The human audit also provides limited support for the reliability of comparisons between methods. These limitations leave me unconvinced that the benchmark's additional contribution is sufficiently validated.

Referenced papers
Questions for authors
  1. What do the authors see as the main technical or system-design contribution of VoxPolyMem beyond AFA, M3-Agent, and M2A? In particular, what new insight comes from the integration, and how is it supported by the evaluation?
  2. What is the substantive advance of EG-GRPO over existing methods that reward useful evidence or discourage repeated retrieval, such as Mem-T and LeTS? How do the reported results establish the importance of that advance beyond showing that the full training recipe improves performance?
  3. What does VoxPolyBench allow us to learn about long-term spoken group memory that existing group-memory and speaker-aware dialogue benchmarks do not? How does the current validation establish this additional value?
4 comments
AuthorsOpenReview ↗

Response to Reviewer MNkR (1/3): W1 & Q1

W1 & Q1: System-design contribution and the role of interaction memory

Thank you for highlighting the explicit speaker-message-addressee relations. VoxPolyMem uses interaction structure as a shared foundation for memory formation and agentic retrieval. This addresses a central difficulty in multi-party conversations: the speaker, addressee, and person an attribute concerns can differ, and these distinctions must survive memory abstraction and subsequent use.

Consider this illustrative exchange:

Bob: "I quite like Thai food."

Alice, addressing Bob: "I remember you're allergic to peanuts, so shall we avoid dishes with peanuts today?"

Bob expresses his own preference, Alice supplies information about Bob's allergy, and avoiding peanut dishes is Alice's proposal. Organizing memory solely by speaker can miss third-party information about Bob; assigning the entire message to its addressee can conflate attributes with proposals.

During memory formation, interaction memory grounds subsequent fact and profile extraction. Online speaker identification and voiceprint updates maintain participant identities, while contextual addressee inference establishes directed links between participants and messages. Fact and profile extraction uses these interaction records to resolve references and interpret participant roles, retaining roles, timestamps, and source references. Higher-level memories remain traceable to the original interactions.

During retrieval, the agent decides how to use these relations. Given the question, asker identity, and evidence already retrieved, the policy selects memory layers and retrieval tools and decides whether speaker or addressee conditions are useful. For Bob's dietary restrictions, relevant evidence may come from Alice, so restricting retrieval to Bob's own utterances would be inappropriate. For a question about whom someone advised, interaction-role conditions may be useful. The policy can combine semantic, keyword, and cross-modal retrieval, then rewrite queries and retrieve additional evidence when needed.

Our design insight is that interaction structure informs both how participant-related memories are formed and how supporting evidence is accessed. Connecting these stages helps use information distributed across different speakers in a shared conversation history; participant relations influence the construction and selection of evidence throughout the pipeline.

The interaction ablation provides targeted evidence. It retains message content and speaker associations while removing addressee-side links and the retrieval operations that depend on them. Both configurations use the same w/o RL setting. The breakdown below follows the category taxonomy preserved with this ablation run. Scores are mean per-question scores multiplied by 100.

CategoryFull interaction relationsWithout interaction relationsGain
Retrieval and reasoning (RR)85.4783.701.77
Evolution and conflict (EC)90.8388.951.88
Persona (P)75.9270.515.41
Interaction attribution (IA)85.9772.4713.50
Overall (Table 3)84.078.65.4

Interaction attribution and persona improve by 13.50 and 5.41 points, respectively, with larger gains than in the other categories. This pattern supports the design motivation: beyond retaining speaker attribution, interaction relations and their use improve access to and differentiation of participant-related information. These gains are already present without RL optimization.

Additional comparisons. We compare AFA and IGMem and add the independently developed EverMemBench.

MethodVoxPolyBenchMem-GalleryH2HMem-MultiEverMemBench
AFA37.647.539.944.3
IGMem58.275.570.753.9
VoxPolyMem85.089.674.467.8

Scores are multiplied by 100; the original three benchmarks retain the main-paper reference runs. VoxPolyMem exceeds IGMem by 26.8, 14.1, 3.7, and 13.9 points, respectively, across multi-party, dyadic multimodal, and independent dialogue settings. These comparisons assess the complete system; the interaction ablation above tests addressee-side relations and access while retaining speaker attribution. EverMemBench category results are discussed in W3/Q3.

Regarding metadata, equivalent relation fields and access operations can express the same logical structure. This ablation evaluates interaction relations and their use, rather than the underlying storage format. We will clarify this comparison and how interaction structure connects memory formation with agentic retrieval.

AuthorsOpenReview ↗

Response to Reviewer MNkR (2/3): W2 & Q2

W2 & Q2: Contributions of evidence-gain credit assignment and ranking rewards

Thank you for asking us to clarify EG-GRPO's methodological contribution and the roles of its individual components. Mem-T and LeTS provide important foundations for process rewards, branching retrieval, and reducing repeated retrieval. EG-GRPO addresses two specific problems in multi-round memory retrieval: assigning credit to actions that recover currently missing evidence, and prioritizing supporting evidence within a limited context. We added a reward-design comparison to distinguish the gains of the round-wise training scheme from the additional contribution of ranking.

1. State-conditioned credit for complementary evidence. An action's utility depends on what has already been retrieved. Evidence useful in the first round may add nothing when retrieved again later. EG-GRPO maps results from different memory layers to source turns, measures newly acquired supporting evidence after fusion, deduplication, and context selection, and compares candidate actions starting from the same retrieval state. If evidence A is already available but B is missing, retrieving B receives direct credit, whereas retrieving A again receives no additional coverage reward. This supplies a training signal for layer selection, tool choice, and query rewriting that reflects the current information gap.

2. Ranking-aware evidence prioritization. Results with the same evidence coverage can place supporting evidence at different positions. The ranking term encourages supporting evidence to appear earlier, so training considers both coverage and ordering within the selected context. This targets more effective evidence organization under limited context and retrieval budgets.

The comparison below reports mean answer scores and evidence recall multiplied by 100. Rounds denotes average retrieval rounds per question, excluding answer generation. VPB, MG, and H2H denote VoxPolyBench, Mem-Gallery, and H2HMem-Multi, respectively.

ConfigurationVPB ScoreVPB RecallVPB RoundsMG ScoreMG RecallMG RoundsH2H ScoreH2H Rounds
Terminal-coverage GRPO83.186.71.486.489.71.571.61.5
EG-GRPO w/o ranking84.886.91.488.193.21.4——
Full EG-GRPO85.287.51.289.693.61.374.51.2

The round-wise training scheme improves performance even without ranking. EG-GRPO w/o ranking exceeds Terminal-coverage GRPO by 1.7 and 1.7 score points on VoxPolyBench and Mem-Gallery. Thus, even when evidence coverage supplies the supervision, training against current information gaps yields different outcomes from assigning trajectory-level credit based on final cumulative coverage. This comparison evaluates the round-wise reward and credit-assignment scheme jointly.

Ranking provides a further quality-efficiency gain. With the remaining settings held fixed, adding the ranking term improves scores by 0.4 and 1.5 points on VoxPolyBench and Mem-Gallery, while reducing average retrieval rounds by approximately 14.3% and 7.1%, respectively. Evidence recall also increases by 0.6 and 0.4 points on VPB and MG. These results directly address whether ranking adds value beyond coverage alone: it improves answer quality and evidence coverage while requiring fewer retrieval rounds.

Overall, Full EG-GRPO exceeds Terminal-coverage GRPO by 2.1, 3.2, and 2.9 score points, with approximately 14.3%, 13.3%, and 20.0% fewer retrieval rounds. The practical contribution is therefore more effective acquisition and organization of complementary evidence within a fixed retrieval budget. We will incorporate these comparisons into the revision and distinguish the evidence supporting the round-wise training scheme from the isolated contribution of ranking. Percentage reductions are calculated from the displayed mean round counts.

3. Consistency across training seeds. We repeated MoT-GRPO and EG-GRPO training with three independent seeds. Scores below are reported as mean ± sample standard deviation, rounded to one decimal. The w/o RL configuration is a fixed reference without policy training.

MethodVoxPolyBenchMem-GalleryH2HMem-Multi
w/o RL84.086.372.1
MoT-GRPO84.0 ± 0.186.9 ± 0.272.4 ± 0.8
EG-GRPO85.2 ± 0.289.6 ± 0.274.5 ± 0.1

EG-GRPO outperforms MoT-GRPO in every matched seed on all three benchmarks, with mean gains of 1.2, 2.7, and 2.1 points. On H2HMem-Multi, its standard deviation is 0.1, compared with 0.8 for MoT-GRPO. These results support repeatable gains across the evaluated seeds, complementing the component ablation above.

AuthorsOpenReview ↗

Response to Reviewer MNkR (3/3): W3 & Q3

W3 & Q3: Benchmark contribution and validation

Thank you for recognizing the value of connecting persistent voice identity with long-term group memory. We clarify this joint evaluation and provide evidence from acoustic conditions, independent dialogue distributions, and human auditing.

1. Joint evaluation: linking identity to information attribution and use across sessions.

GroupMemBench, H2HMem, and PACHAT advance group memory, multi-party multimodal memory, and persona-aware spoken dialogue. VoxPolyBench jointly evaluates persistent voice identity, cross-session fact updates, personalized QA, and interaction attribution. It examines whether systems distinguish the speaker, asker, addressee, and person a fact concerns, and preserve these relationships when answering later questions.

This setting has diagnostic value: one participant's attributes may be expressed by another, while commitments and preferences may change across sessions. As reported in W1/Q1, removing addressee-side relations and associated retrieval operations, while retaining message content and speaker attribution, reduces persona and interaction-attribution scores by 5.4 and 13.5 points. VoxPolyBench thus exposes the capability gap between retaining speaker associations and using interaction relations for long-term memory QA.

2. Acoustic validation: online identity maintenance under overlap and real recordings.

Beyond the IEMOCAP and AMI experiments in the appendix, we evaluated AMI Meeting recordings with natural overlap and the same five VoxPolyBench cases under controlled 10% and 20% overlap. The table reports speaker-ID accuracy (%).

Cumulative configurationVoxPolyBench: 10% overlapVoxPolyBench: 20% overlapAMI: with overlap
Static voiceprint prototypes84.379.365.4
+ Online EMA95.492.285.3
+ Acoustic identity consolidation96.994.886.3

Increasing overlap from 10% to 20% reduces full-module accuracy by approximately 2.1 points, versus 5.1 points for static prototypes. On AMI, the full module improves over static prototypes by 20.9 points. These results extend the evidence for online identity maintenance to overlapping speech and human recordings.

Extending this evaluation to full long-term memory QA on real recordings requires jointly verified cross-session identities, fact attribution and updates, personalized questions, and multi-hop supporting evidence. Existing audio resources have limited joint coverage of these annotations, whose construction requires substantial manual annotation and cross-checking. We plan to incorporate a manually verified real-recording subset for complete memory QA.

3. Independent dialogue distributions: additional EverMemBench evaluation.

We additionally compared AFA, IGMem, and VoxPolyMem on independently developed EverMemBench. Scores are multiplied by 100.

MethodFine-grained recallMemory awarenessUser-profile understandingOverall
AFA10.566.146.944.3
IGMem26.176.346.953.9
VoxPolyMem38.589.462.367.8

VoxPolyMem exceeds IGMem by 13.9 points overall, with category gains of 12.4, 13.1, and 15.5 points. Together with the Mem-Gallery and H2HMem-Multi comparisons in W1/Q1, these results support effectiveness across independent dialogue distributions and different memory tasks.

4. Human validation: dataset quality checks and answer-score auditing.

Dataset construction includes human review of event plans and human/LLM checks of event consistency, participant relations, question clarity, and evidence sufficiency. Correctable issues are revised; QA pairs with unresolved ambiguity or insufficient supporting evidence are removed.

For answer scoring, we sampled 100 responses each from the complete w/o RL evaluations on VoxPolyBench and Mem-Gallery, covering native task categories and proportionally preserving the full result pools' judge-score distributions. Human and LLM assessments used the same five-level rubric.

BenchmarkResponsesLLM meanHuman mean
VoxPolyBench10083.385.0
Mem-Gallery10086.087.3
Overall20084.686.1

Scores are multiplied by 100. Human means are slightly higher on both benchmarks, with an overall gap of 1.5 points. This audit provides a human reference at the aggregate-score level, complementing construction checks, independent-benchmark comparisons, and real-recording identity experiments.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you for your time and constructive feedback. We have addressed your comments in our rebuttal and updated the manuscript with additional experiments and clarifications.

As the discussion period is drawing to a close, we would appreciate any further feedback you may have. If you have any questions, we would be happy to discuss them.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer eVrY (1/3)

Thank you for your thoughtful comments. We address the asker-identity concern in W1 and Q1 first, followed by the distribution of improvements across task categories.

W1-1 & Q1: Are the baselines given the asker's identity?

"Do the baselines receive the identity of the querying participant?"

Thank you for raising this concern about evaluation fairness. We clarify that, for persona questions, all baselines were provided with the ground-truth asker identity at query time, whereas VoxPolyMem identifies the asker through voiceprint matching. Thus, the baselines were not disadvantaged by lacking the asker's identity, and the performance gap cannot be attributed to this missing information. We will explicitly document this setting in the revised manuscript.

Knowing the asker's identity at query time does not, however, guarantee that relevant historical information has been correctly associated with that participant. Personalized information may originate from the participant's own utterances or from statements and commitments made by others. For example, Alex tells Emma, "I booked your follow-up appointment for 3 p.m. tomorrow." When Emma later asks, "What time is my appointment tomorrow?", the relevant evidence comes from Alex's utterance but concerns Emma. Answering correctly requires connecting the speaker, addressee, and participant to whom the fact applies. Providing the asker's identity, or retrieving only the asker's own utterances, is insufficient. VoxPolyMem explicitly preserves these interaction relations to support such evidence association.

Historical speaker association also depends on each method's processing pipeline. For example, M3-Agent uses its own voiceprint-based speaker identification procedure. In our experiments, we observed cases where it assigned multiple identities to the same participant, fragmenting the participant's historical memories. This fragmentation may make it harder to retrieve the relevant evidence even when the ground-truth asker identity is provided at query time.

W1-2: Why are the gains larger on persona and interaction attribution?

"Roughly 85 percent of the 23.6-point gap over M2A comes from the persona and interaction-attribution subsets."

Retrieval/reasoning and evolution/conflict questions in VoxPolyBench primarily assess event retrieval, temporal information integration, and state updates. They place less emphasis on properties specific to multi-party interactions, although they still require associating facts with the correct speakers. For baselines without speaker-identification capabilities, we provide ground-truth speaker labels for the dialogue history, reducing the effect of identity-recognition errors on these tasks. Under this setting, VoxPolyMem still exceeds the strongest baseline in these two categories by 1.7 and 4.8 points, respectively. Its larger gains on persona and interaction attribution are consistent with the capabilities targeted by interaction-aware memory.

We further evaluate generalization across different interaction settings using public benchmarks. H2HMem-Multi evaluates multimodal memory in multi-party conversations, whereas Mem-Gallery evaluates multimodal memory in dyadic conversations. VoxPolyMem achieves scores of 74.4 and 89.6, respectively, exceeding the strongest evaluated public memory baseline by more than eight points on each benchmark. These results support its applicability beyond VoxPolyBench and across both multi-party and dyadic settings. Despite explicitly modeling identity and interaction attribution, VoxPolyMem also achieves the best performance among the compared methods on the dyadic benchmark, showing that its benefits are not confined to tasks dominated by multi-party interaction attributes.

AuthorsOpenReview ↗

Response to Reviewer eVrY (3/3): W3 and Q2

W3.1 / Q2: RL gains and stability across training seeds

Thank you for suggesting a multi-seed evaluation. We conducted independent training runs with three random seeds (40, 41, and 42). Below, QA scores are reported as means and sample standard deviations. The configuration without RL serves as a fixed reference.

MethodVoxPolyBenchMem-GalleryH2HMem-Multi
w/o RL84.0086.3072.10
MoT-GRPO84.00 ± 0.1086.93 ± 0.1572.40 ± 0.79
EG-GRPO85.17 ± 0.1589.63 ± 0.1574.50 ± 0.10

EG-GRPO improves over MoT-GRPO by 1.17, 2.70, and 2.10 percentage points, respectively, and achieves higher QA scores in every matched training seed on all three datasets.

The configuration without RL already achieves strong performance through hierarchical memory and multi-round retrieval. EG-GRPO further improves answer quality while reducing average retrieval rounds:

MethodVoxPolyBenchMem-GalleryH2HMem-Multi
w/o RL1.601.701.40
MoT-GRPO1.431.731.60
EG-GRPO1.201.301.20

Compared with the configuration without RL, EG-GRPO reduces average retrieval rounds by 25.0%, 23.5%, and 14.3%, respectively. Its benefit therefore extends beyond QA scores: the learned policy obtains useful evidence with fewer retrieval rounds.

W3.2: Judge dependence and interpretation of Evidence Recall

To examine sensitivity to the choice of judge, we additionally evaluated the newly trained policies using GPT-4.1 and Gemini 3.8 Flash. The main comparative conclusions remain consistent across these judges. We will report the results separately for each judge in the revision.

We use QA score as the primary evaluation metric and Evidence Recall to measure how well the final retained memory context covers annotated supporting evidence. Sufficient evidence retrieval provides an important basis for reliable answers, so we evaluate both retrieval quality and answer quality.

Compared with MoT-GRPO, EG-GRPO increases mean Evidence Recall from 86.43 to 87.50 on VoxPolyBench and from 91.50 to 93.60 on Mem-Gallery. These improvements accompany higher QA scores and fewer retrieval rounds, showing improvements in supporting-evidence acquisition and downstream answer quality with fewer retrieval rounds.

W3.3: Training hyperparameters

We provide the key training hyperparameters below and will include the complete configuration in the revised appendix.

Clipping coefficient εKL coefficient βLearning rateBatch sizeCandidate actions K
0.20.028 × 10⁻⁷88

W3.4: Measured memory construction and retrieval latency

We measured latency for streaming memory construction and multi-round retrieval. Interaction memory first records utterances, speaker identities, and timestamps from streaming ASR. Profile and episodic memory are then updated in parallel. Addressees are inferred from context during episodic memory extraction and linked to the source utterances and participant nodes within the same operation, without requiring a separate LLM call for addressee identification.

ComponentMeasured latencyMeasurement scope
Speaker identification and voiceprint memory update0.038 s per audio segment, on averageVoiceprint extraction, matching, EMA update, and speaker confirmation
Audio to interaction memory0.87 sIncludes streaming ASR
Profile and episodic memory availability3.291 s on average;Time from the end of an utterance until its content becomes available in the corresponding memory; both branches run in parallel
Multi-round retrieval3.40 s on average1.2 retrieval rounds on average;

The writing pipeline makes new information available progressively. Interaction memory receives new utterances first, while profile and episodic memory are updated asynchronously. Access to newly recorded conversation content therefore does not require waiting for all derived memories to finish updating. On the retrieval side, EG-GRPO reduces the average number of retrieval rounds, with a measured average retrieval latency of 3.40 seconds.

AuthorsOpenReview ↗

Summary of Rebuttal and Manuscript Revisions

We thank the reviewers for their constructive feedback. We have revised the manuscript to strengthen the empirical evaluation and clarify the contributions of VoxPolyMem. The changes are highlighted in blue in the revised manuscript. Below, we summarize the main revisions in four areas.

Interaction-aware memory. We clarify how speaker and addressee relations support fact and profile construction. An interaction-relation ablation shows gains of 5.4 points on persona questions and 13.5 points on interaction attribution.

Retrieval policy optimization. We add a ranking-reward ablation and repeat MoT-GRPO and EG-GRPO training with three seeds. EG-GRPO improves mean answer scores over MoT-GRPO by 1.2, 2.7, and 2.1 points on VoxPolyBench, Mem-Gallery, and H2HMem-Multi, respectively, while using fewer retrieval rounds.

Broader evaluation. We add comparisons with AFA and IGMem and evaluate on EverMemBench. We also evaluate speaker association on AMI recordings with natural overlap and on five VoxPolyBench cases with controlled overlap. VoxPolyMem achieves speaker association accuracy of 96.9% and 94.8% at 10% and 20% overlap, respectively, and 86.3% on AMI.

Practicality and evaluation protocol. We report online memory-update latency, retrieval latency and token usage, and memory-writing cost. We also clarify the training sources, the history-disjoint evaluation setting, and the interpretation of source-evidence recall.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you for your time and constructive feedback. We have addressed your comments in our rebuttal and updated the manuscript with additional experiments and clarifications.

As the discussion period is drawing to a close, we would appreciate any further feedback you may have. If you have any questions, we would be happy to discuss them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.