acceptodds
Desk rejected at ICLR 2027

PERSIST: Who-What-When Memory Across Sessions for Full-Duplex Spoken Dialogue

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern voice assistants may be shared by multiple users and should be able to answer questions about earlier conversations such as “When did I originally plan to leave?” or adapt their behavior to individual users based on past interactions. This requires more than retrieving a topically similar passage: the assistant must identify the current speaker, recover the relevant past state, and distinguish it from later revisions. We present PERSIST, a persistent memory system for multi-session, multi-speaker spoken dialogue that explicitly models Who, What, and When. PERSIST structures cross-session histories into readable event records and retrieves them with a 3W joint scoring mechanism that combines semantic content, acoustic speaker identity, and temporal state. For real-time full-duplex interaction, PERSIST further reuses intermediate representations from the dialogue backbone, avoiding query-audio re-encoding and reducing retrieval latency from 578.42 ms to 7.03 ms. We also introduce SpokenTrace, a diagnostic benchmark that factorizes evaluation along memory tasks and speaker-query types, exposing failures in recall, speaker attribution, and temporal-state tracking. On SpokenTrace, PERSIST achieves 85.08% end-to-end task accuracy and improves all-support EM@3 from 49.01% with BGE-large to 82.10%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Full-duplex dialogue, Persistent conversational memory, Speech language models, Speaker-aware retrieval, Episodic memory, Multi-session dialogue

Related papers

Discussion (0)

Sign in to comment.