- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the lack of consistency in end-to-end Speech LLMs during multi-turn spoken dialogue. The authors attribute this to two bottlenecks. On the architecture side, the speech decoder only receives the LLM hidden states of the current turn, so cross-turn information must be retained entirely by the LLM backbone. On the data side, most existing speech corpora are single-turn, and the available multi-turn datasets are biased toward short, highly colloquial everyday conversations.
The paper’s main contribution is a data synthesis pipeline for building multi-turn spoken dialogues that are semantically coherent and speaker-consistent. The pipeline also annotates each user turn with paralinguistic labels and corresponding response criteria. In addition, the authors propose SpeechAnchor, a lightweight module that lets the speech decoder directly leverage cross-turn speech history. They apply the framework to two speech models, MiMo-Audio and Kimi-Audio, and report performance gains across multiple multi-turn conversation benchmarks.
Critical strengths
-
The paper formulates multi-turn consistency along three clear axes: semantic, paralinguistic, and speaker-style. This formulation aligns with the evaluation dimensions of current benchmarks and offers a unified perspective for future work.
-
The proposed module is simple and cost-efficient, adding only marginal inference overhead. The experimental setup is thorough: the module is applied to two distinct backbones and evaluated on several multi-turn and single-turn benchmarks. The ablation studies are also comprehensive, covering both the architecture and data components as well as the training curriculum.
Critical weaknesses
-
The related work cites several speech LLMs designed specifically for multi-turn conversation (e.g., SLAM-Omni, Audio Flamingo 3, and OmniChat), but none of them is included as a baseline. Without these comparisons, it is difficult to judge how SpeechAnchor compares with existing multi-turn approaches.
-
The ablation in Table 5 shows that removing the SpeechAnchor module causes only a marginal drop in performance. This suggests that most of the improvement comes from the synthesized training data rather than the proposed architecture, which weakens the contribution of the module itself.
-
The quality of the synthetic training data is not sufficiently verified. The authors only report UTMOS scores and speaker-embedding similarity. They do not verify whether the instruct TTS models actually render the specified paralinguistic cues, such as emotion and intonation, in the synthesized speech.
Questions for authors
-
Why are the baselines not the same across the three multi-turn benchmarks? For example, GPT-4o-Audio appears only in the SpeechRole results, and Qwen2.5-Omni-7B is missing from the VoiceAssistant-Eval results.
-
The authors claim that SpeechAnchor reduces the modeling burden on the LLM backbone and gives the speech decoder richer speech-related contextual information. From a representation perspective, what information does the anchor signal actually add? In other words, how do and in Eq. (9) differ? An analysis such as probing these representations for speaker identity or emotion would help support this claim.
-
The eight metrics in Table 3 are hard to follow at first glance. It would help if the authors grouped them by the three consistency axes (semantic, paralinguistic, and speaker-style).
5 comments
Q2: About the contribution of SpeechAnchor module.
A2: We acknowledge that the architectural gain is smaller than that from the synthesized data. However, the two components are complementary: the data provides explicit supervision for semantic, paralinguistic, and speaker-style consistency, while SpeechAnchor aggregates the corresponding information from speech history to exploit this supervision more effectively. Its consistent gains over the -Architecture variant across all MTalk-Bench dimensions directly confirm its effectiveness.
Q3: About the quality validation of synthesized training data.
A3: We thank the reviewer for raising this important concern. As detailed in our response to Reviewer h7sa's Q1, we conducted an additional human listening evaluation to directly assess whether the Instruct TTS models successfully realized the specified paralinguistic cues, including emotion, speaking rate, emphasis, and intonation. The evaluation achieved a mean rating of 4.15/5.0, with a Krippendorff's Alpha of 0.72, demonstrating effective realization of the intended paralinguistic attributes.
Furthermore, we also considered other potential concerns regarding the quality of the synthesized training data, including whether the synthesized speech faithfully preserves the input text, whether it contains omissions, repetitions, or mispronunciations, and whether the generated paralinguistic annotations and corresponding responses are contextually appropriate. To address these concerns, we conducted additional evaluations of transcription fidelity, pronunciation accuracy, and contextual paralinguistic consistency, respectively, as detailed in our response to Reviewer h7sa's Q1.
Overall, these additional evaluations demonstrate that our TTS synthesis pipeline not only effectively realizes the intended paralinguistic cues but also preserves textual fidelity, pronunciation accuracy, and contextual consistency. We will incorporate these evaluation protocols and results into the revised manuscript.
Q4: About inconsistent baseline coverage across benchmarks.
A4: Thank you for pointing this out. The initial results followed benchmark-specific baseline sets and evaluation settings, rather than a unified comparison set. We apologize for the lack of clarity and have supplemented the missing evaluations, consolidating the overall results across all three benchmarks in the table below.
| Model | MTalk-Bench (Avg.) | VoiceAssistant-Eval (Final) | SpeechRole-Eval (Overall) |
|---|---|---|---|
| Kimi-Audio-7B | 58.17 | 24.00 | 65.30 |
| Step-Audio (130B) | 58.25 | 43.15 | 75.60 |
| LLaMA-Omni2-14B | 58.93 | 41.00 | 74.10 |
| MiMo-Audio-7B | 59.46 | 39.50 | 82.60 |
| Qwen2.5-Omni-7B | 61.12 | 55.74 | 68.60 |
| GPT-4o-Audio | 69.50 | 63.98 | 86.90 |
| SpeechAnchor (Kimi) | 61.89 | 25.90 | 70.20 |
| SpeechAnchor (MiMo) | 65.32 | 42.70 | 88.80 |
The results show that SpeechAnchor (MiMo) consistently improves over its corresponding backbone across all three benchmarks. Although it does not achieve the highest absolute score on every benchmark, this is partly attributable to differences in model scale, architecture, and training data. Nevertheless, considering the three benchmarks collectively, SpeechAnchor (MiMo) achieves the best performance among models of comparable scale on the majority of benchmarks, demonstrating its consistent effectiveness across diverse evaluation settings.
Q5: About the representation and interpretability of anchor signals.
We thank the reviewer for this insightful question. We clarify the information contributed by the anchor signal from both mechanistic and experimental perspectives.
-
As defined in Eq. (7)–(9), denotes the original speech-decoder layer output, whereas is an additional history-conditioned representation retrieved by SpeechAnchor. Thus, combines the original decoder representation with complementary information from historical speech states.
-
This additional information covers three dimensions: (1) Semantic information: Historical speech states encode the content of previous utterances, enabling the decoder to retain contextual information. (2) Paralinguistic information: Since these states originate from speech representations, they retain acoustic cues such as emotion and intonation, together with their evolution across turns. (3) Speaker-related information: Our synthesis pipeline maintains consistent reference speech within each dialogue, allowing historical speech states to carry speaker-related characteristics. Through recurrent state updates, SpeechAnchor integrates these complementary aspects of speech history and injects the resulting representation into the decoder.
-
Although these factors may already be encoded in the LLM hidden states, relying solely on the LLM backbone to simultaneously track and integrate them imposes a heavy modeling burden, particularly when relevant context is distributed across speech and text modalities. SpeechAnchor alleviates this burden by explicitly modeling speech-modality history and providing the speech decoder with a dedicated pathway to access these complementary representations.
-
Our ablation study (Table 5) shows that removing SpeechAnchor reduces the MTalk-Bench average score from 65.32 to 63.68, with consistent declines across all three dimensions. Figure 2 further shows higher cross-turn speaker-embedding cosine similarity with SpeechAnchor.
These results support our explanation that the anchor signal provides complementary historical information across semantic, paralinguistic, and speaker-style dimensions, enriching the representations received by the speech decoder and facilitating more effective multi-turn consistency modeling.
Q6: About the organization of evaluation metrics.
A6: Thank you for this helpful suggestion. We agree that grouping the eight metrics according to the three consistency dimensions would make Table 3 easier to understand. We have reorganized the metrics as follows:
-
Semantic Consistency (IA, CC, KC): IA (Instruction Adherence) evaluates adherence to task instructions and character identity; CC (Conversational Coherence) assesses logical consistency with dialogue context; KC (Knowledge Consistency) measures consistency with established character knowledge and background.
-
Paralinguistic Consistency (EA, PC): EA (Emotion Appropriateness) evaluates whether emotional expressions align with the dialogue context and character state; PC (Prosodic Consistency) assesses whether prosody aligns with the intended speaking style and remains consistent throughout the dialogue.
-
Speaker-Style Consistency (SF, SN, PeC): SF (Speech Fluency) evaluates fluency and appropriate pacing; SN (Speech Naturalness) assesses how natural and human-like the generated speech sounds; PeC (Personality Consistency) measures consistency with the character's personality traits.
We will revised Table 3 and the corresponding explanations in the main text accordingly.
Thanks again for your careful and valuable comments. We hope our response addresses your concerns. If our responses answer your questions well and reassure your concerns, we would appreciate it if you could reassess our work and increase the rating.
Q1: About comparisons with existing multi-turn SLMs.
A1: Thank you for this valuable suggestion. We agree that comparisons with existing multi-turn SLMs are important for assessing the effectiveness of SpeechAnchor.
Related Work and Baseline Selection. Existing multi-turn SLMs explore different approaches to dialogue history modeling, including textual history modeling in SLAM-Omni, acoustic retrieval in Audio Flamingo 3, multimodal feature fusion in OmniChat, and dedicated alignment in InteractiveOmni and Audio Flamingo 3.
Multi-turn Evaluation. We evaluate both baselines on the same three multi-turn speech-to-speech benchmarks using their official inference configurations and the original benchmark evaluation protocols.
Table 1: Results on MTalk-Bench.
SpeechAnchor achieves the best overall performance across all three tracks, with particularly substantial gains in Paralinguistic and Ambient. This demonstrates its effectiveness in modeling cross-turn paralinguistic and acoustic information.
Table 2: Results on VoiceAssistant-Eval.
SpeechAnchor achieves the highest Final score, with the best Content and Speech performance while maintaining comparable text-speech Consistency. Notably, this metric measures agreement between textual and spoken outputs rather than multi-turn dialogue consistency. These results demonstrate improved response and speech quality without compromising text-speech agreement.
Table 3: Results on SpeechRole-Eval.
SpeechAnchor achieves the best overall performance and outperforms both baselines across all eight metrics, with particularly strong gains in Prosodic Consistency, Emotion Appropriateness, and Personality Consistency. These results demonstrate its ability to preserve speaker style while adapting to evolving paralinguistic cues, supported by its speech-history modeling and two-stage training strategy.
Single-turn Evaluation. We further evaluate the models on four general-purpose SpokenQA benchmarks to examine whether multi-turn consistency enhancement affects general speech capabilities.
Table 4: Results on General SpokenQA Benchmarks.
SpeechAnchor maintains competitive performance across all four benchmarks, demonstrating that the proposed multi-turn enhancement does not compromise general spoken question answering and instruction-following capabilities. This is consistent with our design: the additional anchor pathway provides historical context to the speech decoder without changing the core LLM architecture, while the two-stage training strategy focuses on consistency modeling while preserving general representations.
Overall, these comparisons demonstrate that SpeechAnchor improves multi-turn spoken dialogue consistency across semantic, paralinguistic, and speaker-style dimensions while preserving general speech capabilities.