acceptodds
Under review as a conference paper at ICLR 2027

VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models

Abstract

Spoken conversational systems must recover information from prior interactions (i.e., memory), yet relevant information in speech extends beyond what was said to who said it, how it was spoken, and what was audible, information that exists only in the audio signal and cannot be recovered from a transcript. Beyond what to remember, memory also demands diverse operations: retrieving a single fact, integrating evidence across turns, tracking an evolving state. Real interactions further unfold across sessions, meaning information accumulates across distinct episodes rather than a single continuous recording. Existing benchmarks fall short on all three dimensions: they focus primarily on lexical content, adopt limited and ad hoc memory operations, and treat memory as a single-session problem. We argue that principled memory evaluation requires jointly characterizing the acoustic evidence to be retained and the operations applied to it, and introduce a taxonomy along these two axes. Building on this taxonomy, we present VoxMem: 3,196 evaluation instances over 34,743 spoken sessions (177 hours) crossing four acoustic evidence types (speech semantics, speaker identity, paralinguistic cues, environmental sound) with four memory operations (information extraction, multi-session reasoning, temporal tracking, and answer refusal), grounded in multi-session histories and stratified across context budgets from 8K to 64K tokens. Evaluating 15 LALMs, no model exceeds 40% at 32K. Models retain what was said far better than who said it, how, or what was audible, a gap that widens for complex operations, grows with history length, and manifests as qualitatively distinct failure modes across evidence types. VoxMem aims to provide a foundation to measure and drive progress on the full scope of spoken conversational memory. Code and data are available at https://anonymous.4open.science/r/voxmem-FEA0.

Primary area:
datasets and benchmarks
Keywords:
Multimodal, Benchmark, Converstational Memory, Large Audio Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4sLyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces VoxMem, a benchmark for evaluating memory in large audio language models across multiple sessions. It covers semantic content, speaker identity, paralinguistic cues, and environmental sounds. The results show that models struggle with information beyond semantic content, especially as conversation histories grow longer.

Critical strengths
  1. Memory is an important topic, and memory in large audio language models remains underexplored.
  2. The benchmark goes beyond semantic content to cover speaker identity, paralinguistic cues, and environmental sounds. I appreciate this broader coverage.
Critical weaknesses
  1. I found the paper difficult to follow. It introduces many terms, making it hard to grasp for my first and second pass of reading.
  2. The data are synthesized through a complicated pipeline, which may introduce bias. It is also unclear how well the data reflect real conversations.
  3. The paper lacks cascaded baselines or toplines, such as various speech, audio encoders, ASR systems paired with an LLM.
Questions for authors

The authors should motivate more on how the memory operations are defined. Why these four? and why there four are important?

Reviewer_MA4NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VoxMem evaluates spoken conversational memory across four evidence types and four memory operations. It contains 799 questions, each evaluated at four history lengths, using synthetic speech and constructed multi-session conversations. Experiments on 15 audio language models show weaker performance on non-lexical information and declining accuracy as histories grow.

Critical strengths
  1. The benchmark covers speaker identity, vocal delivery, and environmental sounds beyond transcript-level recall, making it relevant to spoken assistants.

  2. Keeping questions and evidence fixed across history lengths provides a useful controlled comparison. The broad model coverage and transcript-only checks further support its evaluation value.

Critical weaknesses
  1. Limited realism and independent validation. All user speech is synthesized, with fixed reference voices, controlled speaking styles, and environmental sounds mixed at a fixed SNR. These choices simplify real conversational variability. Without natural-recording validation or reported human listening checks, the reliability and real-world relevance of the benchmark remain uncertain.

  2. Perception and memory failures are not sufficiently separated. Even the shortest histories contain substantial audio, and no evidence-only audio baseline is reported. Low scores may therefore reflect difficulty recognizing the acoustic cues rather than retaining them. Transcript-only comparisons establish acoustic necessity but do not resolve this issue, limiting the paper’s conclusions about memory.

Questions for authors
  1. Was any human listening validation conducted? If so, what were the sample size and agreement on cue recognizability and answer correctness?

  2. Do you have existing results using only the answer-relevant audio to distinguish basic perception failures from failures caused by longer histories?

  3. Was any subset checked against natural recordings or an alternative speech synthesis setup to assess dependence on the construction pipeline?

Reviewer_vyp7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VoxMem evaluates memory over multi-session spoken conversations, crossing four evidence types: speech semantics, speaker identity, paralinguistic cues, and environmental sound, with information extraction, multi-session reasoning, temporal tracking, and answer refusal. It contains 799 questions instantiated at four reference history budgets, yielding 3,196 evaluation instances. Histories combine answer-bearing sessions, competing distractors, and unrelated fillers, with synthesized user audio and textual assistant responses.

The contributions are the evaluation taxonomy, benchmark construction, and an analysis of 15 audio-language models. No evaluated model exceeds 40% overall accuracy at 32K, with substantially weaker performance on non-lexical acoustic evidence. Validation includes model-based quality control, transcript ablations, a second scoring judge, and paired bootstrap estimates for context scaling.

Critical strengths
  1. The benchmark targets consequential information missing from transcripts. Speaker–fact associations, vocal delivery, and environmental sounds are important to spoken interaction but are poorly represented by purely lexical memory evaluations. Organizing these evidence types across multiple operations provides a useful framework for identifying weaknesses that aggregate conversational scores can obscure.

  2. The construction provides useful experimental controls and extensive documentation. Reusing questions and intended evidence across history budgets reduces variation in question difficulty. Competing sessions discourage simple topic matching, while transcript ablations provide evidence that acoustic information matters. The appendices further strengthen the contribution through explicit scoring rubrics, judge cross-validation, and paired uncertainty estimates.

Critical weaknesses
  1. The evaluation does not adequately distinguish memory failure from perception and reasoning failure. Models receive the complete history at query time, and low final-answer accuracy can arise from failing to recognize a cue, retrieve its session, bind it correctly, or perform the requested operation. The paper does not report independent human audio validation or per-model oracle-evidence and oracle-acoustic-annotation baselines. These omissions are consequential because the conclusions concern retaining and updating acoustic information, whereas the experiments establish difficulty on the combined task. Similarly, Appendix E’s response-based error attribution cannot reliably identify the underlying failure mechanism: an incorrect count may reflect either missed evidence or faulty counting. A retrieval or structured acoustic-memory baseline would also help establish how much difficulty comes from the full-history interface.

  2. The controlled history comparison does not isolate length, and several conclusions exceed the statistical evidence. Appendix B.6 increases same-key distractors from up to 1 to 2, 4, and 8 across budgets while interleaving additional sessions. Consequently, length, interference, and evidence distances change together. The experiment supports degradation under this combined expansion procedure, but not a distinct causal effect of history length. Furthermore, separate confidence intervals for within-type drops do not establish differences in decay rates between evidence types. The abstract’s widening-gap claim also needs clarification: Table 18 shows that the absolute semantic–acoustic percentage-point gaps shrink from 8K to 64K.

Questions for authors
  1. Are the acoustic cues and reference answers reliable? The paper describes model-based quality checks, but I would like to understand how well people can recognize the synthesized cues and agree with the reference answers. Do you have human validation results, or results from giving each model only the sessions containing the relevant evidence? These would help distinguish difficulty perceiving the audio from difficulty remembering or retrieving it.
  2. What drives the decline as histories get longer? Longer histories also introduce more same-key distractors and change the distances between relevant evidence and the query. Could you clarify how much of the decline comes from additional context versus increased interference? A comparison with fixed distractor counts and controlled evidence positions would help support the interpretation of the scaling results.
  3. Could you clarify which evaluation release produced the reported results? The public scoring implementation excludes missing verdicts from the accuracy denominator, while Appendix C.4 states that unparseable verdicts and generation failures count as incorrect. The dataset documentation also identifies four questions whose histories are not strictly nested (VoxMem code release, 2026; VoxMem dataset documentation, 2026). Were these differences present in the version used for the paper? Please explain how they were handled and whether they affect the reported scores or paired comparisons. I recognize that the current public release may differ from the submission-time version.
  4. Could you clarify the contribution relative to prior work and the checks for source overlap? EnvMem already studies multi-turn acoustic memory and its failure mechanisms (Xiao et al., 2026). SD-Eval evaluates spoken interaction beyond lexical content (Ao et al., 2024), while RAIL includes a structured framework for auditory memory (Jin et al., 2026). A more explicit comparison would help identify what VoxMem adds. Separately, did you check whether the ESC-50 recordings, VCTK voice references, or InstructS2S content overlap with disclosed training data for the evaluated models or their encoders? Such overlap would not by itself demonstrate answer leakage, but documenting it would help distinguish familiarity with source material from performance on new memory associations.

References:

  • Ao, J., et al. (2024). SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words. arXiv:2406.13340.
  • Jin, H., et al. (2026). RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark. arXiv:2606.11260.
  • Xiao, Y., et al. (2026). Why Can’t They Remember? Uncovering Representation and Retrieval Bottlenecks in Multi-Turn Acoustic Memory. arXiv:2605.27039.
  • VoxMem code release (2026). Accuracy aggregation implementation, commit 2e3ef97dd3fb5585c77bcaa361c348ebc9f1da29. Source code.
  • VoxMem dataset documentation (2026). AudioMemory/voxmembench dataset card. Dataset documentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.