acceptodds
Under review as a conference paper at ICLR 2027

VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction

Abstract

Real-time conversational systems require relevant memory within limited context and response-time budgets. We introduce VoiceMem, a streaming memory layer with factual and persona-affective retrieval branches, shared evidence identifiers, and optional entity attribution. An evolving schema–entity index adapts to recurring queries, while per-turn maintenance separately records immediate reactions and model-proposed persona claims. A streaming controller runs the same retrieval code before speech endpoint confirmation, overlapping preparation with the utterance so that only unfinished final ranking remains on the memory-critical path when preparation is complete. VoiceMem supports interchangeable backends and includes a teacher-supervised adaptation workflow, ChatMem-400K, and the audio-grounded ChatMem-Bench. Evaluations cover factual, persona, and audio-memory quality, context efficiency, component contributions, and three backend integrations. VoiceMem improves on reported factual and persona baselines, including a 21.43-point gain over Full-Context on LoCoMo, and exceeds MemOS by 14.20 percentage points on ChatMem-Bench. At the default LoCoMo operating point (K=5), the same text-evaluation setting achieves 90.4 with 430 memory tokens and 134 ms mean memory-system latency without streaming overlap. Streaming further hides preparation behind speech, connecting evidence organization with response-time memory preparation while separating memory management from the response model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Memory system, speech language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7RAKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper divides the memory retrieval into two branches, i.e., the left brain for the factual, and the right brain for the persona-affective. In this way, they can be constructed and updated in different ways and with different frequencies. In addition, the retrieval processes are conducted in a streaming way. Memories are retrieved before an utterance ends, thus greatly reducing the memory-system latency. Experiments show great response quality and low latency.

Critical strengths
  • This paper introduces an interesting framework with dual-brain memory for voice interaction.
  • This paper introduces a streaming controller to prepare memory from partial transcripts.
  • Extensive experiments demonstrate the effectiveness of the memory retrieval framework and the response quality.
Critical weaknesses
  • This paper is not easy to follow, and should be better organized. For example, in section 3, the descriptions of Eq. 1-2 make the Eq. 3 harder to understand with different notations without clear explanations.
  • Notations should be defined before they are used. For example, N_v^{micro} and N_s^{macro} in line 161 are not defined.
  • Some important details are missing. Using partial transcripts involves much more computation, thus the frequency to check the partial information should be clarified. In Eq. 10, the improved quality may benefit from the prompt engineering, thus the construction of the prompt should be explained.
Questions for authors

Please see the weaknesses.

1 comment

I have read the rebuttal, and would like keep my positive ratings.

Reviewer_MRF9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a memory structure for a conversation agent, especially a voice conversation agent such that one part of the memory infers traits regarding the person who is speaking, and the other files the facts about that person. Also another claim is that the system is able to gather facts about the person as the person speaks, which makes the approach specifically attuned for speech, as opposed to text.

Overall, the approach is in spirit similar to RAG, but the paper proposes a disentangled approach for the aforementioned aspects. which seem to benefit the experimental results, in terms of fact retrieval (table 1). The advantage seem less clear though for personality retrieval (table 2)

Critical strengths
  • The results in table 1 suggests an advantage towards the proposed approach, that is the fact indexing system of the paper seems to provide an edge. The proposed model (voicemem) does not beat the full context baseline on all three datasets (on Memora the average scores seem to indicate that voicemem is slightly behind) -- though this is fair for voicemem as it uses a smaller memory size.
Critical weaknesses
  • The improvements seem to be less substantial on persona memory. Especially when broken down in terms of each dataset. Especially when performance is broken down in terms of each dataset.

  • I found the paper difficult to read. I feel at some parts it really requires investigation to get the main message. For instance, for each table, it would help to include the average for each dataset, otherwise it is hard to understand the ranking. Also the introduction and abstract can help a more clear and directed narrative.

  • I cannot seem to locate an experiment that measures what the streaming fact retrieval brings. There is a table (Table 9, Appendix H.3.1), but the advantage seems to be marginal.

Questions for authors
  • What is the input to the LLM-judge prompts? Is it the final answer, or are you giving the retrieved facts as well? I could not locate the prompt if you are giving it somewhere.

  • It is unclear how reliable the LLM-judge is. Did you do any comparison of agreement with human opinions? Also it can help to use a second judge model. This is an important point since the arguments for the paper hinge on the LLM-judge scores.

Reviewer_i3kCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VoiceMem, a memory layer for real-time spoken dialogue systems. VoiceMem splits memory into two specialized branches. The factual branch routes queries through an evolving schema–entity index, which adds new schemas when recurring query patterns appear. The persona-affective branch stores user preferences, attitudes, and emotional reactions as claim-level records with timestamped evidence. A streaming controller begins memory preparation from partial ASR transcripts before the end of speech is detected, so that only final ranking remains after the user stops speaking. The memory layer is decoupled from both the storage backend and the response model. The authors also release a synthetic training corpus (ChatMem-400K), an audio-grounded memory benchmark (ChatMem-Bench), and a teacher-supervised adaptation of Qwen3-Omni. Experiments on factual, persona, and audio-grounded memory benchmarks report improvements over a range of existing memory systems with a compact memory context. Additional experiments show transfer across three storage backends and reduced post-endpoint retrieval delay under streaming.

Critical strengths

S1. A complete, working system for an under-explored problem. The paper integrates factual routing, a persona/affect store, speaker attribution, and a streaming controller into a single memory layer designed for spoken dialogue. The appendix documents implementation details unusually thoroughly, including thresholds, write ordering, timeout behavior, and full prompts, which makes the system considerably easier to reproduce than most memory-system papers.

S2. Broad evaluation and a potentially useful benchmark. The comparison covers nine memory systems plus Full-Context across seven benchmarks spanning factual, persona, and audio-grounded memory. ChatMem-Bench adds categories that existing memory benchmarks lack, namely background-sound recall, acoustic-scene inference, and multi-speaker memory, and could benefit the spoken-dialogue community if released.

Critical weaknesses

W1. Nearly all quality evidence is text-only, despite the voice-centric framing. Although VoiceMem is motivated as a memory layer for voice interaction, six of the seven benchmarks are text corpora. Even on ChatMem-Bench, all systems in the main comparison (Table 3) receive transcripts with textualized audio events, scenes, and emotion labels rather than audio. Each session also has a single fixed speaker, with multi-speaker information written directly in text. The only audio-input row scores lower and also changes the response model, so it does not isolate modality. As a result, the voice-specific claims remain essentially unvalidated, including acoustic-cue memory, speaker attribution, and robustness to ASR and speaker-identification errors.

W2. The closest prior work is not compared. The authors acknowledge that DualMem already separates factual and persona memory and that StreamRAG already overlaps retrieval with ongoing speech, yet neither is included as a baseline. Since all compared systems use a single store, it is unclear whether VoiceMem’s gains come from the dual-branch design, which is not new, or from its own contributions such as schema–entity routing and schema emergence. This omission is hard to justify given that several other baselines were run through local re-implementations of their published mechanisms.

Questions for authors

Q1. Appendix H.5 reports that VoiceMem uses 430 memory tokens at K=5, compared with 1,899 for EverMemOS and 541–6,956 across baselines. Were the baselines also evaluated at K=5, or at their own main configurations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.