- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Vox-Infinity to evaluate how spoken language models retain and use information in extended histories. It contains three subsets: Ultra Multi-Turn Dialogues based on SpokenWOZ, Personal Monologues reconstructed from GigaSpeech, and Beyond-Semantic Dialogues based on IEMOCAP and emotion-controlled TTS. These target fine-grained information retrieval, long-form speech understanding, and memory for emotional cues, respectively.
Critical strengths
-
The paper brings multi-turn precise retrieval, long single-speaker speech, and emotional memory into one evaluation framework, covering needs beyond short-clip audio QA.
-
Beyond-Semantic contains 2,758 natural samples and 2,622 semantically controlled samples. Synthesizing different emotions from neutral text is a meaningful design for testing whether acoustic cues provide additional information.
Critical weaknesses
-
The paper repeatedly claims to be the first benchmark for long-context understanding in spoken language models and treats the absence of answer-provenance annotations in prior work as a central distinction. However, marking ContextDialog as “x” under Prov. in Table 1 requires correction or explanation. Its §3.1 explicitly requests a supporting utterance for each QA pair and validates the evidence; its appendix describes this as metadata for locating information in dialogue history.
-
BLAB [1] also already studies emotion and localization in long audio. The more defensible additions here are therefore the combination of three task types, stratification by evidence distance, and the controlled emotional-memory design, rather than a broad claim of being the first long-audio understanding evaluation.
-
Different bins contain different questions and sessions. There is no paired control that keeps the question and evidence content fixed while varying only evidence position or distractor-history length, and no oracle-audio condition supplying only the supporting segments.
[1] Ahia, Orevaoghene, et al. "Blab: Brutally long audio bench." arXiv preprint arXiv:2505.03054 (2025).
Questions for authors
-
Why is ContextDialog marked “x” under Prov. in Table 1? How do the annotations here differ from its supporting utterance in timestamps, multiple evidence spans, and evidence necessity?
-
Could the authors report no-history, supporting-segments-only oracle audio, and full-history conditions? For a fixed set of QA pairs, does the trend persist when evidence content is held constant and only position or distractor duration changes?
-
Which models had max_position_embeddings modified, and what were their native limits and positional settings? How many audio or text tokens did each model actually receive, and how were chunking, speakers, and turn boundaries handled? Please provide checkpoint/API versions, evaluation dates, prompts, and generation settings, and verify the trend within natively supported ranges.
4 comments
Response to Reviewer BLvd (2/2): Supporting-evidence controls
W3/Q2: No-history, supporting-turn-only, and full-history controls.
We randomly sampled 100 QA examples from each of four subsets: Ultra Multi-Turn Dialogues (UMT), Personal Monologues (PM), natural emotion, and semantically controlled synthetic emotion. We compared three conditions on the same QA examples and reference answers:
- No history: the current question without historical audio.
- Supporting turns only: only the annotated utterance audio needed to answer the question, without transcripts or reference answers.
- Full history: the complete audio history associated with the question.
Semantic tasks.
| Subset | Model | No history | Supporting turns only | Full history |
|---|---|---|---|---|
| PM | Qwen3-Omni | 12% | 84% | 72% |
| PM | Audio Flamingo Next | 8% | 82% | 38% |
| PM | GLM-4-Voice | 6% | 76% | 24% |
| UMT | Qwen3-Omni | 2% | 86% | 78% |
| UMT | Audio Flamingo Next | 0% | 78% | 40% |
| UMT | GLM-4-Voice | 0% | 72% | 38% |
In all six comparisons, supplying the supporting turns substantially improves accuracy over no history. Relative to supporting turns only, full-history accuracy falls by 12, 44, and 52 percentage points on PM and by 8, 38, and 34 points on UMT.
Because the questions and reference answers are held fixed, these differences cannot be attributed to changing the question composition between conditions. They show an additional challenge in locating and using the same evidence within a complete audio history, compared with receiving the supporting turns directly. This provides within-question validation complementary to the distance curves.
Supporting-turn input still requires accurate speech perception and answer production. For example, some UMT errors involve reproducing exact digit sequences such as telephone numbers. A small number of no-history answers can also draw on prior knowledge: a question about neural networks' poor performance in the 1980s can be answered in terms of limited computation and data without hearing the particular monologue.
Emotion tasks.
| Subset | Model | No history | Supporting turns only | Full history |
|---|---|---|---|---|
| Natural emotion | Qwen3-Omni | 48% | 56% | 66% |
| Natural emotion | Audio Flamingo 3 | 24% | 28% | 34% |
| Natural emotion | Audio Flamingo Next | 24% | 30% | 32% |
| Synthetic emotion | Qwen3-Omni | 28% | 48% | 40% |
| Synthetic emotion | Audio Flamingo 3 | 10% | 30% | 18% |
| Synthetic emotion | Audio Flamingo Next | 18% | 28% | 22% |
For natural emotion, full history improves accuracy over supporting turns alone by 10, 6, and 2 percentage points. This is consistent with surrounding dialogue providing situational and semantic information that helps interpret the target utterance's emotion.
Semantically controlled synthetic emotion shows a different pattern. Supporting turns improve accuracy over no history by 20, 20, and 10 points, whereas full history reduces accuracy relative to supporting turns by 8, 12, and 6 points. Here, the target emotion is conveyed through the acoustic expression of neutral text, so the surrounding semantics need not provide the same emotional cues as in natural dialogue.
These results reveal the task-dependent role of history: semantic tasks and controlled emotion show an additional challenge in using evidence within the full history, while natural emotion can also benefit from the interpretive information provided by surrounding dialogue.
Thank you for the detailed response. I suggest that the authors characterize the evidence-distance curves as descriptive trends and include the context extension settings for each model in the revised manuscript. The authors’ rebuttal has adequately addressed my main concerns, so I am raising my score from 2 to 3.
Follow-up to Reviewer BLvd
Thank you for your follow-up and for reassessing our submission. In the revision, we will describe the evidence-distance curves as descriptive trends and report the context extension settings for each model. We appreciate your constructive feedback.
Response to Reviewer BLvd (1/2): Contributions and evaluation setup
Thank you for your constructive feedback. Part 1 addresses W1/Q1, W2, and Q3; Part 2 reports W3/Q2 controls.
W1/Q1: ContextDialog and supporting-evidence annotations.
Thank you for pointing out ContextDialog's supporting-utterance annotations. We will update its annotation entry in Table 1.
Vox-Infinity extends the evaluation scope and capability coverage. As summarized in Table 1, ContextDialog averages 68.9 seconds and 10.60 turns per dialogue, whereas our Ultra Multi-Turn Dialogues (UMT) and Beyond-Semantic Dialogues (BSD) average 35.89 and 55.23 turns, respectively. Personal Monologues (PM) additionally covers long continuous speech. Together, these settings assess precise retrieval across many turns, long-form understanding and reasoning, and memory for emotional cues.
Our annotations link answers to utterances or segments with temporal boundaries and can associate multiple supporting segments with a question. QA pairs and their evidence are manually checked for consistency. We use the interval from the earliest supporting evidence to the query to organize evaluation, allowing questions from the same history to have different evidence distances. These annotations support fine-grained analysis within the broader benchmark design.
W2: Relation to BLAB and contribution.
BLAB studies emotion and localization in long audio. Vox-Infinity establishes a systematic evaluation framework for long dialogues and long-form speech while examining how history representations affect model capabilities. Its design has three connected aspects:
Complementary long-history capabilities. Ultra multi-turn dialogues, personal monologues, and emotional dialogues assess fine-grained retrieval, cross-segment understanding and reasoning, and paralinguistic emotional memory, covering both frequent turn exchanges and long continuous speech.
Systematic comparison of text and audio histories. We compare the two representations on the same tasks, examining their differences in preserving precise semantic information and acoustic emotional cues, and how these differences vary with evidence distance. This provides empirical guidance for history representation in spoken dialogue systems.
Semantically controlled acoustic emotional memory. Alongside natural emotional dialogues, we synthesize different emotional expressions from neutral text to reduce direct lexical cues to the answer. Combined with text/audio history comparisons, this more specifically tests whether models retain and use acoustic emotional cues from past utterances.
Supporting-evidence annotations and distance-based grouping enable fine-grained analysis of these capabilities. We will emphasize this overall positioning in the abstract and related work and clarify the relationship to prior benchmarks.
Q3: Model versions, positional limits, and input organization.
Limits below 32,768 were raised to 32,768 for UMT/BSD. PM contains longer continuous speech histories, producing longer encoded input sequences, so limits below 65,536 were raised to 65,536 for PM. Higher original limits were retained. Thus, GLM and MiMo were extended for all three subsets, and Audio Flamingo 3 only for PM. We will add this per-model, per-subset table to Section 4.1.1.
Utterances, roles, and turn boundaries. The dataset explicitly annotates turn boundaries. We input history as successive messages following these boundaries, preserving chronological order and dialogue roles. Each message contains its turn's audio in the audio-history condition or its transcript in the text-history condition. Turn segmentation and role assignments are supplied by the data, rather than inferred by the model.
Questions and prompts. After the history, a final human/user message contains the original question, without additional task instructions, reasoning guidance, or answer hints. Messages follow each model's corresponding chat template.