- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates long-term memory in multi-party, cross-session spoken dialogues and introduces VoxPolyMem, a system that integrates online speaker recognition, interaction dynamics, factual memory, and participant profiling, utilizing a retrieval agent to manage memory layer selection, tool usage, and query rewriting. The authors further propose EG-GRPO, a training method that uses the coverage and ranking of newly acquired supporting evidence per turn as rewards for the retrieval strategy. Additionally, the paper presents VoxPolyBench, a dataset comprising 18 scenarios, 176 sessions, 18.9 hours of synthetic speech, and 1,527 question-answer pairs. Experimental results demonstrate significant performance gains on this benchmark as well as on two public multimodal memory benchmarks, with ablation studies analyzing the roles of memory structure, iterative retrieval, and strategy training.
Critical strengths
The problem formulation is practically significant. Long-term, multi-party conversations require tracking both "who said what" and "to whom it was said"; simply storing facts can easily lead to confusion regarding information attribution. This paper explicitly models participants and interaction relationships—utilizing them for retrieval filtering—representing a promising direction for research.
The system's performance gains do not rely solely on the authors' custom benchmark. The method also demonstrates improvements on two public multimodal memory benchmarks, with ablation studies confirming the contributions of interaction relationships, memory hierarchy, and iterative retrieval. These results substantiate the framework's practical value.
Critical weaknesses
In Table 2, VoxPolyMem scores 85.0 on the self-constructed benchmark, yet the version without RL already achieves 84.0. The RL gains on the two public benchmarks are 3.3 and 2.3 points, respectively. Consequently, the substantial lead over external baselines was largely established prior to the introduction of EG-GRPO. While the authors explicitly report this fact, it implies that the overall system's superiority cannot serve as direct evidence of the new training algorithm's significance. Existing work like Mem-T (Yue et al., 2026) already integrates hierarchical memory, iterative retrieval, and MoT-GRPO, providing process supervision through tree-based sampling, node-level evidence rewards, and outcome backpropagation. Thus, the concept of "shifting from sparse terminal rewards to step-wise credit assignment" already has direct precedents. The specific novel contributions of this paper lie in deduplicating newly added evidence, incorporating ranking rewards, and comparing candidate actions within the same state.
Although the authors do compare their method against MoT-GRPO in Table 4, that comparison simultaneously alters both the reward composition and the credit assignment mechanism, making it impossible to determine which specific design is effective. The study lacks independent ablation studies comparing new versus cumulative evidence, coverage versus ranking rewards, and intra-state grouping versus other grouping methods; furthermore, it reports neither training variance across runs nor a standardized computational budget. The advantage over MoT-GRPO on the self-constructed benchmark is a mere 0.9 points, and the available evidence is insufficient to determine whether this difference is stable. A more fundamental issue is that the reward and Recall metrics measure source ID coverage rather than whether the retained content actually contains the evidence required to answer the question. Appendices B.3 and C.2 clarify that a single fact or profile record may contribute multiple source utterance IDs via refer_id, yet the answering model receives a compressed record rather than the full content of those source utterances. The fact that a record is linked to a supporting utterance does not guarantee that it retains the specific information from that utterance needed to answer the current question.
The description of the train-test split is limited to the number of QA pairs. Table 5 divides VoxPolyBench into 200 training questions and 1,327 test questions but fails to specify whether they originate from mutually exclusive scenes, participants, or conversation histories. The same lack of detail applies to the data split for Mem-Gallery. If training and test questions share the same history, the results primarily demonstrate adaptation to new questions within a known memory bank, rather than supporting generalization to unfamiliar groups or novel interaction histories. The paper currently lacks sufficient information to distinguish between these scenarios. The speech setup is tightly controlled: participants use consistent reference voices across sessions, and the synthesis process incorporates re-synthesis and transcription consistency checks, resulting in a final Word Error Rate (WER) of 2.14%. While these design choices ensure data quality, they also minimize the acoustic variability typical of real-world speech scenarios. The IEMOCAP and AMI experiments in Appendix D.1 rely on known utterance boundaries and evaluate only speaker tracking; they do not examine how identity errors in actual recordings propagate to interaction relationships, fact attribution, participant profiling, and final answers. Accuracy metrics for the recognition module on real recordings cannot substitute for a comprehensive evaluation of the memory-based QA system using actual speech input.
AFA employs ECAPA-TDNN speaker recognition to route speech queries to the corresponding user's cross-session memory and dynamic profile. The current paper lacks a direct comparison that isolates the value of the newly introduced interaction structures. AFA focuses on individual user memories, whereas this paper addresses shared multi-party histories—scenarios that are not equivalent; this distinction necessitates a comparison to determine exactly how "identity routing plus individual memory" differs from "identity routing plus shared interaction relationships."
Questions for authors
see weakness
5 comments
Response to Reviewer rWy3 (2/3): Evidence validity and history-disjoint evaluation
Thank you for raising the concerns about evidence validity and generalization. We clarify both below.
1. Source-evidence coverage and answer quality. The retrieval policy selects interaction, fact, or profile memory, and the answer model receives the actual records from the selected layers. The refer_id links map facts and profiles to their source utterances, providing a common evidence unit across memory granularities and preventing repeated credit for the same source across layers or rounds. “Expansion” in Appendix B.3 refers to this source-ID mapping for reward computation, not replacement of every retrieved record with the full source utterance.
Source coverage measures provenance overlap with annotated supporting evidence; it does not by itself guarantee that a record preserves every detail needed for the current question. Accordingly, inference-time sufficiency checks operate on the retrieved content and can trigger further retrieval, while QA scores independently assess generated answers rather than ID matches. Across three training seeds, EG-GRPO improves mean QA scores over MoT-GRPO by 1.2, 2.7, and 2.1 points on VoxPolyBench, Mem-Gallery, and H2HMem-Multi, respectively. The gains therefore extend to answer quality. We will explicitly define the metric as source-evidence coverage and distinguish it from content sufficiency.
2. Training and test conversation histories are disjoint. Policy training uses the sources listed below. Test questions come from different conversation histories, rather than a question-level split within a shared memory bank.
| Benchmark | RL training sources | Training QA pairs | Test-history separation |
|---|---|---|---|
| VoxPolyBench | G003, G010 | 200 | Histories disjoint from the training scenarios |
| Mem-Gallery | Home-Repair, Music-Dance, Real-Estate | 200 | Histories disjoint from the training topics |
| H2HMem-Multi | None | 0 | Evaluation only |
These experiments thus evaluate policy transfer to conversation histories and memory banks unseen during training, rather than adaptation to new questions about known histories. We will add the scenario/topic identifiers and history-disjoint split protocol to the revision and reconcile the reported training counts accordingly.
Response to Reviewer rWy3 (3/3): Real speech and shared interaction memory
Thank you for raising the questions about real-speech evaluation and the value of shared interaction memory. We provide additional acoustic tests and baseline comparisons below.
1. Real recordings and overlapping speech. Beyond the IEMOCAP and AMI experiments in Appendix D.1, we evaluated AMI Meeting recordings with natural overlap and the same five VoxPolyBench cases under controlled 10% and 20% overlap. The table reports speaker-ID accuracy (%).
| Cumulative configuration | VoxPolyBench: 10% overlap | VoxPolyBench: 20% overlap | AMI: with overlap |
|---|---|---|---|
| Static voiceprint prototypes | 84.3 | 79.3 | 65.4 |
| + Online EMA updates | 95.4 | 92.2 | 85.3 |
| + Acoustic identity consolidation | 96.9 | 94.8 | 86.3 |
The complete module improves over static prototypes by 20.9 points on AMI. Increasing controlled overlap from 10% to 20% reduces its accuracy by 2.1 points, compared with 5.1 points for static prototypes. These results support online identity maintenance under overlapping speech and real recordings.
These experiments evaluate identity association. Measuring how identity errors propagate to fact attribution, profiles, and final QA additionally requires real recordings with jointly verified cross-session identities, interaction relations, fact updates, and question-specific supporting evidence. Existing resources have limited joint coverage of these annotations, whose construction requires substantial manual annotation and verification. We plan to incorporate a manually verified real-recording memory-QA subset to evaluate the complete pipeline.
2. AFA, IGMem, and the contribution of shared interaction relations. We added comparisons with identity-aware memory method AFA [1] and multimodal multi-party memory method IGMem [2], including an evaluation on independently developed EverMemBench. Scores are mean answer scores multiplied by 100; VoxPolyMem retains the main-paper reference runs on the original three benchmarks.
| Method | VoxPolyBench | Mem-Gallery | H2HMem-Multi | EverMemBench |
|---|---|---|---|---|
| AFA [1] | 37.6 | 47.5 | 39.9 | 44.3 |
| IGMem [2] | 58.2 | 75.5 | 70.7 | 53.9 |
| VoxPolyMem | 85.0 | 89.6 | 74.4 | 67.8 |
VoxPolyMem exceeds IGMem, the stronger additional baseline, by 26.8, 14.1, 3.7, and 13.9 points, respectively. The gains span multi-party conversations, dyadic multimodal conversations, and an independently constructed benchmark.
Identity routing establishes who is asking; shared interaction relations further help establish whom information concerns. For example, Alex tells Emma, “I remember you're allergic to peanuts, so shall we avoid dishes with peanuts today?” Alex is the speaker, but the allergy concerns Emma. VoxPolyMem uses these relations to construct facts and profiles, while allowing the retrieval policy to apply speaker or addressee conditions when useful.
To examine this contribution directly, we retain message content and speaker attribution while removing addressee-side links and the retrieval operations that depend on them. Persona and interaction-attribution scores decrease from 75.9 and 86.0 to 70.5 and 72.5, respectively; both configurations use w/o RL. The baseline comparisons assess the complete system, while this ablation tests interaction relations and their associated access mechanisms beyond speaker-only attribution. Together, they provide complementary evidence for shared interaction memory. We will incorporate these additional results and clarify their evaluation scope in the revision. Differences are computed before rounding.
[1] AFA: Identity-Aware Memory for Preventing Persona Confusion in Multi-User Dialogue. https://arxiv.org/abs/2604.25022
[2] IGMem: Interaction-Grounded Memory for Multimodal Multi-Party Conversations. https://openreview.net/forum?id=3lSas96zVV
Response to Reviewer rWy3 (1/3): EG-GRPO contribution and stability
Response to the concerns about EG-GRPO's contribution, reward ablations, and training variability
Thank you for asking us to distinguish the contributions of memory organization and policy training. EG-GRPO improves supporting-evidence acquisition and organization within the existing memory system, increasing answer quality while reducing retrieval rounds. We added reward-design comparisons and three independent training seeds to examine the contribution of ranking and the repeatability of the training gains.
1. Methodological contribution: complementary evidence acquisition and ranking.
Mem-T has explored node-level evidence rewards, tree-based sampling, and outcome backpropagation. EG-GRPO focuses on supporting evidence still missing from the current context: after fusion, deduplication, and context selection, only newly acquired supporting evidence receives additional coverage reward. The ranking term encourages supporting evidence to appear earlier within the limited context.
For example, if evidence A is already available but B is missing, retrieving A again through another memory layer receives no additional coverage reward, whereas recovering B receives direct credit. This aligns training with the information gaps that subsequent retrieval rounds need to resolve.
2. Reward-design comparisons.
VPB, MG, and H2H denote VoxPolyBench, Mem-Gallery, and H2HMem-Multi. Score and Recall are multiplied by 100. Rounds counts retrieval rounds per question, excluding answer generation. Terminal-coverage GRPO retains the paper's reference-run results.
Without ranking, EG-GRPO still exceeds terminal cumulative-coverage training by 1.7 and 1.7 score points on VoxPolyBench and Mem-Gallery, respectively, supporting the effectiveness of the round-wise training scheme.
With the remaining settings held fixed, adding ranking further improves scores by 0.4 and 1.5 points on VoxPolyBench and Mem-Gallery, while reducing average retrieval rounds by approximately 14.3% and 7.1%. Evidence recall also increases by 0.6 and 0.4 points on VoxPolyBench and Mem-Gallery. This directly demonstrates the additional value of ranking beyond coverage alone: evidence coverage and answer quality improve with fewer retrieval rounds.
Overall, Full EG-GRPO exceeds Terminal-coverage GRPO by 2.1, 3.2, and 2.9 score points, with approximately 14.3%, 13.3%, and 20.0% fewer retrieval rounds.
3. Repeatability across three training seeds.
Trained methods report mean ± sample standard deviation across three seeds; w/o RL is a fixed reference configuration. EG-GRPO outperforms MoT-GRPO in every matched seed on all three benchmarks, with mean gains of 1.2, 2.7, and 2.1 points. On H2HMem-Multi, its standard deviation is 0.1, compared with 0.8 for MoT-GRPO. Relative to w/o RL, EG-GRPO also reduces mean retrieval rounds from 1.6, 1.7, and 1.4 to 1.2, 1.3, and 1.2.
These experiments support EG-GRPO through reward ablation and repeated training: ranking provides measurable additional benefits, and the complete method consistently improves answer quality across the evaluated seeds while using fewer retrieval rounds. We will incorporate these results into the revision and distinguish memory-system gains from the further improvements due to policy optimization. Score differences are computed before display rounding; round reductions use the reported mean round counts.