acceptodds
Under review as a conference paper at ICLR 2027

SpeechAnchor: Enhancing Speech Large Language Models for Consistent Multi-Turn Spoken Dialogue

Abstract

End-to-end Speech Large Language Models (SLMs) have shown strong potential for natural voice interaction by directly modeling speech-to-speech communication. However, practical voice interaction is inherently multi-turn, requiring SLMs to maintain semantic coherence, adapt to evolving paralinguistic cues, and preserve speaker style throughout the conversation. Existing SLMs remain limited in multi-turn dialogue because most training corpora are dominated by single-turn conversations, and existing architectures mainly rely on the LLM backbone to track information across dialogue history. To address these challenges, we propose **SpeechAnchor**, a multi-turn consistency enhancement framework for SLMs. SpeechAnchor combines a lightweight module that explicitly models cross-turn speech history for the speech decoder with a consistency-aware data pipeline that constructs semantically coherent, speaker-consistent, and paralinguistically rich multi-turn speech data. Experiments on three multi-turn and four single-turn speech benchmarks show consistent gains in multi-turn consistency without compromising general speech performance. These results establish SpeechAnchor as a practical framework for advancing multi-turn SLMs and provide a unified perspective to guide future research on multi-turn consistency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech Large Language Models, Multi-turn Speech Dialogue, Dialogue Consistency, Paralinguistic Awareness, Speaker Style Preservation, Context-aware Speech Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aSJKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the lack of consistency in end-to-end Speech LLMs during multi-turn spoken dialogue. The authors attribute this to two bottlenecks. On the architecture side, the speech decoder only receives the LLM hidden states of the current turn, so cross-turn information must be retained entirely by the LLM backbone. On the data side, most existing speech corpora are single-turn, and the available multi-turn datasets are biased toward short, highly colloquial everyday conversations.

The paper’s main contribution is a data synthesis pipeline for building multi-turn spoken dialogues that are semantically coherent and speaker-consistent. The pipeline also annotates each user turn with paralinguistic labels and corresponding response criteria. In addition, the authors propose SpeechAnchor, a lightweight module that lets the speech decoder directly leverage cross-turn speech history. They apply the framework to two speech models, MiMo-Audio and Kimi-Audio, and report performance gains across multiple multi-turn conversation benchmarks.

Critical strengths
  1. The paper formulates multi-turn consistency along three clear axes: semantic, paralinguistic, and speaker-style. This formulation aligns with the evaluation dimensions of current benchmarks and offers a unified perspective for future work.

  2. The proposed module is simple and cost-efficient, adding only marginal inference overhead. The experimental setup is thorough: the module is applied to two distinct backbones and evaluated on several multi-turn and single-turn benchmarks. The ablation studies are also comprehensive, covering both the architecture and data components as well as the training curriculum.

Critical weaknesses
  1. The related work cites several speech LLMs designed specifically for multi-turn conversation (e.g., SLAM-Omni, Audio Flamingo 3, and OmniChat), but none of them is included as a baseline. Without these comparisons, it is difficult to judge how SpeechAnchor compares with existing multi-turn approaches.

  2. The ablation in Table 5 shows that removing the SpeechAnchor module causes only a marginal drop in performance. This suggests that most of the improvement comes from the synthesized training data rather than the proposed architecture, which weakens the contribution of the module itself.

  3. The quality of the synthetic training data is not sufficiently verified. The authors only report UTMOS scores and speaker-embedding similarity. They do not verify whether the instruct TTS models actually render the specified paralinguistic cues, such as emotion and intonation, in the synthesized speech.

Questions for authors
  1. Why are the baselines not the same across the three multi-turn benchmarks? For example, GPT-4o-Audio appears only in the SpeechRole results, and Qwen2.5-Omni-7B is missing from the VoiceAssistant-Eval results.

  2. The authors claim that SpeechAnchor reduces the modeling burden on the LLM backbone and gives the speech decoder richer speech-related contextual information. From a representation perspective, what information does the anchor signal actually add? In other words, how do ut,jℓu^{\ell}_{t,j} and u~t,jℓ\tilde{u}^{\ell}_{t,j} in Eq. (9) differ? An analysis such as probing these representations for speaker identity or emotion would help support this claim.

  3. The eight metrics in Table 3 are hard to follow at first glance. It would help if the authors grouped them by the three consistency axes (semantic, paralinguistic, and speaker-style).

5 comments

Q1: About comparisons with existing multi-turn SLMs.

A1: Thank you for this valuable suggestion. We agree that comparisons with existing multi-turn SLMs are important for assessing the effectiveness of SpeechAnchor.

  • Related Work and Baseline Selection. Existing multi-turn SLMs explore different approaches to dialogue history modeling, including textual history modeling in SLAM-Omni, acoustic retrieval in Audio Flamingo 3, multimodal feature fusion in OmniChat, and dedicated alignment in InteractiveOmni and Audio Flamingo 3.

    • We include SLAM-Omni and InteractiveOmni as baselines. We use the official Multi-Round Dialogue (English) checkpoint and inference recipe of SLAM-Omni, and the publicly released 8B checkpoint of InteractiveOmni with its official multi-turn speech generation configuration.
    • We exclude Audio Flamingo 3 because its publicly released implementation supports only text output, despite its architecture describing a streaming TTS module. We also exclude OmniChat because its trained checkpoints and complete inference implementation are not publicly available, preventing reproducible evaluation.
  • Multi-turn Evaluation. We evaluate both baselines on the same three multi-turn speech-to-speech benchmarks using their official inference configurations and the original benchmark evaluation protocols.

    Table 1: Results on MTalk-Bench.

    ModelSemanticParalinguisticAmbientAvg.
    SLAM-Omni (0.5B)52.0048.5047.0049.17
    InteractiveOmni-8B64.0057.5061.5061.00
    SpeechAnchor (MiMo)64.9062.4668.6065.32

    SpeechAnchor achieves the best overall performance across all three tracks, with particularly substantial gains in Paralinguistic and Ambient. This demonstrates its effectiveness in modeling cross-turn paralinguistic and acoustic information.

    Table 2: Results on VoiceAssistant-Eval.

    ModelContentConsistencySpeechFinal
    SLAM-Omni (0.5B)52.5090.5068.0032.31
    InteractiveOmni-8B63.8091.8067.5039.53
    SpeechAnchor (MiMo)65.2091.6071.4042.70

    SpeechAnchor achieves the highest Final score, with the best Content and Speech performance while maintaining comparable text-speech Consistency. Notably, this metric measures agreement between textual and spoken outputs rather than multi-turn dialogue consistency. These results demonstrate improved response and speech quality without compromising text-speech agreement.

    Table 3: Results on SpeechRole-Eval.

    ModelIACCSFSNPCEAPeCKCOverall
    SLAM-Omni (0.5B)70.076.070.072.064.062.065.074.069.1
    InteractiveOmni-8B89.095.079.078.079.084.082.091.084.6
    SpeechAnchor (MiMo)93.197.980.278.285.590.590.494.588.8

    SpeechAnchor achieves the best overall performance and outperforms both baselines across all eight metrics, with particularly strong gains in Prosodic Consistency, Emotion Appropriateness, and Personality Consistency. These results demonstrate its ability to preserve speaker style while adapting to evolving paralinguistic cues, supported by its speech-history modeling and two-stage training strategy.

  • Single-turn Evaluation. We further evaluate the models on four general-purpose SpokenQA benchmarks to examine whether multi-turn consistency enhancement affects general speech capabilities.

    Table 4: Results on General SpokenQA Benchmarks.

    ModelLlama-QWeb-QAlpacaTriviaQA
    SLAM-Omni (0.5B)45.020.04.025.0
    InteractiveOmni-8B71.539.07.542.0
    SpeechAnchor (MiMo)69.737.97.340.7

    SpeechAnchor maintains competitive performance across all four benchmarks, demonstrating that the proposed multi-turn enhancement does not compromise general spoken question answering and instruction-following capabilities. This is consistent with our design: the additional anchor pathway provides historical context to the speech decoder without changing the core LLM architecture, while the two-stage training strategy focuses on consistency modeling while preserving general representations.

Overall, these comparisons demonstrate that SpeechAnchor improves multi-turn spoken dialogue consistency across semantic, paralinguistic, and speaker-style dimensions while preserving general speech capabilities.

Q2: About the contribution of SpeechAnchor module.

A2: We acknowledge that the architectural gain is smaller than that from the synthesized data. However, the two components are complementary: the data provides explicit supervision for semantic, paralinguistic, and speaker-style consistency, while SpeechAnchor aggregates the corresponding information from speech history to exploit this supervision more effectively. Its consistent gains over the -Architecture variant across all MTalk-Bench dimensions directly confirm its effectiveness.

Q3: About the quality validation of synthesized training data.

A3: We thank the reviewer for raising this important concern. As detailed in our response to Reviewer h7sa's Q1, we conducted an additional human listening evaluation to directly assess whether the Instruct TTS models successfully realized the specified paralinguistic cues, including emotion, speaking rate, emphasis, and intonation. The evaluation achieved a mean rating of 4.15/5.0, with a Krippendorff's Alpha of 0.72, demonstrating effective realization of the intended paralinguistic attributes.

Furthermore, we also considered other potential concerns regarding the quality of the synthesized training data, including whether the synthesized speech faithfully preserves the input text, whether it contains omissions, repetitions, or mispronunciations, and whether the generated paralinguistic annotations and corresponding responses are contextually appropriate. To address these concerns, we conducted additional evaluations of transcription fidelity, pronunciation accuracy, and contextual paralinguistic consistency, respectively, as detailed in our response to Reviewer h7sa's Q1.

Overall, these additional evaluations demonstrate that our TTS synthesis pipeline not only effectively realizes the intended paralinguistic cues but also preserves textual fidelity, pronunciation accuracy, and contextual consistency. We will incorporate these evaluation protocols and results into the revised manuscript.

Q4: About inconsistent baseline coverage across benchmarks.

A4: Thank you for pointing this out. The initial results followed benchmark-specific baseline sets and evaluation settings, rather than a unified comparison set. We apologize for the lack of clarity and have supplemented the missing evaluations, consolidating the overall results across all three benchmarks in the table below.

ModelMTalk-Bench (Avg.)VoiceAssistant-Eval (Final)SpeechRole-Eval (Overall)
Kimi-Audio-7B58.1724.0065.30
Step-Audio (130B)58.2543.1575.60
LLaMA-Omni2-14B58.9341.0074.10
MiMo-Audio-7B59.4639.5082.60
Qwen2.5-Omni-7B61.1255.7468.60
GPT-4o-Audio69.5063.9886.90
SpeechAnchor (Kimi)61.8925.9070.20
SpeechAnchor (MiMo)65.3242.7088.80

The results show that SpeechAnchor (MiMo) consistently improves over its corresponding backbone across all three benchmarks. Although it does not achieve the highest absolute score on every benchmark, this is partly attributable to differences in model scale, architecture, and training data. Nevertheless, considering the three benchmarks collectively, SpeechAnchor (MiMo) achieves the best performance among models of comparable scale on the majority of benchmarks, demonstrating its consistent effectiveness across diverse evaluation settings.

Q5: About the representation and interpretability of anchor signals.

We thank the reviewer for this insightful question. We clarify the information contributed by the anchor signal from both mechanistic and experimental perspectives.

  • As defined in Eq. (7)–(9), u~t,jℓ\tilde{\mathbf{u}}_{t,j}^{\ell} denotes the original speech-decoder layer output, whereas Wrt,jW\mathbf{r}_{t,j} is an additional history-conditioned representation retrieved by SpeechAnchor. Thus, ut,jℓ\mathbf{u}_{t,j}^{\ell} combines the original decoder representation with complementary information from historical speech states.

  • This additional information covers three dimensions: (1) Semantic information: Historical speech states encode the content of previous utterances, enabling the decoder to retain contextual information. (2) Paralinguistic information: Since these states originate from speech representations, they retain acoustic cues such as emotion and intonation, together with their evolution across turns. (3) Speaker-related information: Our synthesis pipeline maintains consistent reference speech within each dialogue, allowing historical speech states to carry speaker-related characteristics. Through recurrent state updates, SpeechAnchor integrates these complementary aspects of speech history and injects the resulting representation into the decoder.

  • Although these factors may already be encoded in the LLM hidden states, relying solely on the LLM backbone to simultaneously track and integrate them imposes a heavy modeling burden, particularly when relevant context is distributed across speech and text modalities. SpeechAnchor alleviates this burden by explicitly modeling speech-modality history and providing the speech decoder with a dedicated pathway to access these complementary representations.

  • Our ablation study (Table 5) shows that removing SpeechAnchor reduces the MTalk-Bench average score from 65.32 to 63.68, with consistent declines across all three dimensions. Figure 2 further shows higher cross-turn speaker-embedding cosine similarity with SpeechAnchor.

These results support our explanation that the anchor signal provides complementary historical information across semantic, paralinguistic, and speaker-style dimensions, enriching the representations received by the speech decoder and facilitating more effective multi-turn consistency modeling.

Q6: About the organization of evaluation metrics.

A6: Thank you for this helpful suggestion. We agree that grouping the eight metrics according to the three consistency dimensions would make Table 3 easier to understand. We have reorganized the metrics as follows:

  • Semantic Consistency (IA, CC, KC): IA (Instruction Adherence) evaluates adherence to task instructions and character identity; CC (Conversational Coherence) assesses logical consistency with dialogue context; KC (Knowledge Consistency) measures consistency with established character knowledge and background.

  • Paralinguistic Consistency (EA, PC): EA (Emotion Appropriateness) evaluates whether emotional expressions align with the dialogue context and character state; PC (Prosodic Consistency) assesses whether prosody aligns with the intended speaking style and remains consistent throughout the dialogue.

  • Speaker-Style Consistency (SF, SN, PeC): SF (Speech Fluency) evaluates fluency and appropriate pacing; SN (Speech Naturalness) assesses how natural and human-like the generated speech sounds; PeC (Personality Consistency) measures consistency with the character's personality traits.

We will revised Table 3 and the corresponding explanations in the main text accordingly.

Thanks again for your careful and valuable comments. We hope our response addresses your concerns. If our responses answer your questions well and reassure your concerns, we would appreciate it if you could reassess our work and increase the rating.

Reviewer_4foGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets multi-turn consistency (semantic, paralinguistic, speaker style) in end-to-end speech LLMs. It adds a lightweight Gated DeltaNet module (67M parameters) that aggregates speech-position hidden states from previous turns and injects the result into every layer of the speech decoder. It also proposes a data pipeline that converts Magpie-Ultra multi-turn text dialogues into speech with consistent speakers, plus a paralinguistic subset where LLM-assigned cue labels drive style-controlled TTS and rewritten responses. With two-stage training on MiMo-Audio-7B and Kimi-Audio-7B (100k dialogues, 13.4k h), the method improves MTalk-Bench, VoiceAssistant-Eval, and SpeechRole-Eval over the base models while keeping single-turn SpokenQA roughly unchanged, with ~1.5% inference overhead.

Critical strengths
  1. Task-aligned data construction for multi-turn speech consistency. A major strength is the consistency-aware data synthesis pipeline, particularly its selection of paralinguistically sensitive dialogues and generation of responses conditioned on both dialogue history and acoustic-style cues. Unlike approaches primarily focused on emotion annotations or speech synthesis, this pipeline provides direct supervision for how an assistant should adapt its responses to evolving paralinguistic information. This is well aligned with the target task and represents a practically useful contribution.

  2. Consistent empirical improvements with low architectural overhead. SpeechAnchor improves performance across two different SLM backbones and three multi-turn benchmarks while maintaining comparable general spoken QA performance. The additional Gated DeltaNet pathway incurs only approximately 1.5% full-turn decoding overhead on MiMo-Audio-7B. This favorable performance-efficiency trade-off strengthens the practical relevance of the framework.

Critical weaknesses
  1. Insufficiently controlled comparison between synthesized and existing multi-turn speech datasets. One point that I found particularly noteworthy in the ablation experiment is that, replacing the proposed data with public datasets reduces MTalk-Bench performance to 45.04, considerably below the original MiMo-Audio baseline of 59.46. This raises the possibility of training distribution mismatch or negative transfer, rather than demonstrating an intrinsic advantage of the proposed synthesis strategy. The comparison against existing public datasets does not clearly control for training data volume, dialogue complexity, domain distribution, or speech synthesis characteristics. Since data construction is a central contribution, more controlled comparisons are needed to identify which aspects of the pipeline are responsible for the improvements.

  2. Limited isolation of the architectural contribution and insufficient validation of real-world paralinguistic generalization. Although the Gated DeltaNet-based anchor is conceptually well motivated, the architecture-only improvement is relatively modest (65.32 versus 63.68 on MTalk-Bench). The paper does not compare the proposed module against simpler alternatives, such as attention-based history aggregation or other lightweight recurrent mechanisms, under matched conditions.

Questions for authors
  1. Controlled data comparisons. Could you give a comparison against public multi-turn speech datasets under matched training duration, data volume, and, where feasible, comparable synthesis or preprocessing conditions? Why does the -Data variant perform substantially worse than the original backbone? I wonder why your data is so good.

  2. Architectural necessity. Have you compared Gated DeltaNet against simpler alternatives, such as attention-based aggregation or other recurrent memory modules, under comparable parameter and computational budgets? Could you clarify how recurrent states are maintained or reconstructed across dialogue turns? These details are important for evaluating the specific architectural contribution. A full-scale experiment is quite expensive, so perhaps some lightweight alternative experiments and an analysis of interpretability methods could also support your choice.

6 comments

Q1: About fair dataset comparisons, the -Data performance drop and advantages of our synthesized data.

A1: Thank the reviewer for raising these important concerns.

  • Explaining the Performance Drop of the -Data Variant. Existing public multi-turn dialogue datasets are valuable resources for spoken dialogue modeling. However, they are often biased toward daily conversational scenarios, with short dialogues, simple responses, and highly colloquial expressions. Such data provide limited supervision for complex reasoning and instruction following, which may weaken the model's ability to handle challenging multi-turn interactions and explain the substantial performance drop of the -Data variant.

  • Controlled Comparison with Public Multi-turn Speech Datasets.

    To conduct a controlled comparison, we selected the five public multi-turn speech datasets used in our original -Data ablation: DeepDialogue, IEMOCAP, MultiDialog, DailyTalk, and MELD.

    • Data Preparation. We constructed two datasets under matched speech synthesis conditions:

      • Public-ReTTS: We retained the original dialogue texts, responses, speaker assignments, and available emotion annotations, and re-synthesized all utterances using the same two Instruct TTS models as our pipeline, CosyVoice 3 and Gemini-2.5-Pro-Preview-TTS. Emotion labels were mapped to corresponding TTS style prompts, with unlabeled utterances synthesized in a neutral style. Speaker identities were kept consistent across turns. We applied the same audio preprocessing, quality filtering, and speech encoding pipeline as our original data.
      • Ours-Matched: We sampled complete dialogues from our original Semantic and Paralinguistic Consistency Data, preserving the original 80:20 ratio and all associated supervision. Sampling was performed at the dialogue level, with the total audio duration matched to Public-ReTTS.

      After synthesis and quality filtering, Public-ReTTS contained 50,892 dialogues and 438,741 utterances, totaling approximately 800 hours of speech. Ours-Matched was constructed with the same total audio duration. Both datasets were used to train SpeechAnchor (MiMo) with identical model initialization, architecture and training configurations. Evaluation was conducted on MTalk-Bench.

    • Results and Analysis. The results are summarized below.

      ModelTraining DataSemanticPara.AmbientAvg.
      MiMo-Audio-7BOriginal baseline64.4556.9656.9659.46
      SpeechAnchor (MiMo)Original -Data41.2351.1742.7345.04
      SpeechAnchor (MiMo)Public-ReTTS42.0050.5044.0045.50
      SpeechAnchor (MiMo)Ours-Matched64.5059.6561.5261.89
      SpeechAnchor (MiMo)Ours-Full64.9062.4668.6065.32

      Under matched audio duration, TTS synthesis, and training conditions, Ours-Matched outperforms Public-ReTTS in average score, with consistent gains across all three dimensions. Notably, Public-ReTTS remains close to the original -Data result, suggesting that the performance drop is not primarily attributable to differences in TTS models or acoustic synthesis conditions.

    These results demonstrate that our advantage lies in task-aligned supervision. Unlike public datasets that mainly provide emotion labels or conversational speech, our pipeline explicitly aligns dialogue context, evolving paralinguistic cues, and context-aware responses, improving semantic and paralinguistic consistency.

  • Why Our Synthesized Data Works Better. We build on Magpie-Ultra, a high-quality multi-turn instruction corpus with rich supervision for reasoning and instruction following, diverse challenging scenarios, and relatively detailed responses that better match the backbone's training distribution. We further ensure consistent role-specific speaker identities and construct paralinguistically rich dialogues through context-aware cue annotation and response generation, providing targeted supervision for all three dimensions of multi-turn consistency.

Q2: About architectural gains, comparisons with alternative modules, and explanation of cross-turn recurrent-state management.

A2: Thank the reviewer for these constructive suggestions.

  • Comparison with Alternative History Aggregation Modules.

    • Experimental Setup. We compare Gated DeltaNet with three representative alternatives: Causal Cross-Attention for explicit history retrieval, GRU for vector-based recurrent memory, and DeltaNet without Gating to isolate the contribution of adaptive gating to Delta Rule-based memory updates.

      • All experiments use MiMo-Audio-7B as the backbone, with identical inputs, training data, training strategies, and decoder-side residual injection. All modules receive the same historical speech representations and current hidden state. The additional parameters of Cross-Attention and GRU are matched to Gated DeltaNet (~67.4M), while DeltaNet without Gating retains the same architecture but removes the gating pathway.
      • To reduce training costs, all five groups, including No Anchor, use the same 25% training budget and two-stage training strategy. All models are evaluated on the MTalk-Bench using the same Gemini-2.5-Pro protocol, with computational costs profiled under identical inference settings.
    • Results and Analysis. As shown below, all four explicit history aggregation mechanisms outperform No Anchor, confirming the benefit of introducing a dedicated historical information pathway into the speech decoder.

      ArchitectureSemanticPara.AmbientAvg.
      No Anchor61.4060.2065.5062.37
      Causal Cross-Attention63.2061.3065.8063.43
      GRU62.5061.1065.8063.13
      DeltaNet62.7061.8065.9063.47
      Gated DeltaNet (Ours)62.9062.6066.0063.83
      • Causal Cross-Attention achieves the best Semantic performance, benefiting from direct query-based retrieval over the complete history without fixed-dimensional state compression. In contrast, DeltaNet outperforms GRU across all three dimensions and overall, suggesting that its matrix-based associative memory and content-dependent Delta Rule updates better capture relationships among historical speech representations than conventional vector-based recurrent memory.
      • Gated DeltaNet achieves the best overall performance, particularly on Paralinguistic. Compared with ungated DeltaNet, adaptive gating further improves paralinguistic modeling by complementing Delta Rule updates with flexible control over historical information retention and decay. This is particularly important in multi-turn dialogue, where emotional and prosodic cues may evolve across turns, while speaker identity and stable vocal characteristics should be preserved. Thus, Gated DeltaNet better balances semantic continuity, paralinguistic adaptability, and speaker-style consistency.
      ArchitectureFLOPs/token P50FLOPs/token P90Per-step LatencyFull-turn LatencyGPU Memory P50/P90
      No Anchor1.00×1.00×1.000×1.000×+0% / +0%
      Causal Cross-Attention1.31×1.58×1.035×1.060×+0.9% / +1.4%
      GRU1.01×1.01×1.010×1.012×+0.7% / +0.7%
      DeltaNet w/o Gating1.07×1.07×1.009×1.011×+0.7% / +0.7%
      Gated DeltaNet (Ours)1.09×1.09×1.012×1.015×+0.7% / +0.7%
      • The computational efficiency results are shown in the table above. Overall, Causal Cross-Attention incurs higher computational costs as dialogue history grows, while GRU and DeltaNet offer slightly lower inference costs. Gated DeltaNet achieves a favorable balance between multi-turn performance and computational efficiency.
  • Architectural Contribution. Although the architectural gain is more modest than that of the synthesized data, the two components are complementary: the data provides explicit supervision for semantic, paralinguistic, and speaker-style consistency, while SpeechAnchor aggregates the corresponding information from speech history to exploit this supervision more effectively. Its consistent gains over the -Arch variant across all MTalk-Bench dimensions confirm its contribution.

  • Cross-turn Recurrent-State Management. SpeechAnchor reconstructs its recurrent state at each dialogue turn rather than carrying it over from the previous turn. Specifically, the state is initialized to zero and reconstructed by sequentially processing all preceding speech-related LLM hidden states. Within the current turn, the state is updated autoregressively after each speech position, with the current position incorporated only after its anchor representation is computed, thereby preserving causality. This design maintains a fixed-dimensional recurrent state without persistent cross-turn storage, at the cost of recomputing historical representations at each turn.

Q3: About controlled data comparisons.

A3: We thank the reviewer for this question. As detailed in Q1, under matched audio duration (~800 hours), TTS models, preprocessing, and training configurations, Ours-Matched achieves 61.89 on MTalk-Bench, substantially outperforming Public-ReTTS (45.50). The latter remains close to the original -Data result (45.04), suggesting that our advantage primarily stems from task-aligned supervision, including context-aware paralinguistic annotations, evolving cues, and corresponding responses.

Q4: About architectural necessity.

A4: We thank the reviewer for this suggestion. As detailed in Q2, we compared Gated DeltaNet with Causal Cross-Attention, GRU, and DeltaNet under a reduced 25% training budget. Gated DeltaNet achieves the best average score (63.83), with only 1.2% additional per-step latency and 0.7% GPU memory overhead. Comparisons with DeltaNet further highlight the contribution of adaptive gating in balancing paralinguistic adaptability and speaker-style consistency.

Regarding recurrent-state management, the state is initialized to zero and reconstructed at each dialogue turn by sequentially processing historical speech-related hidden states. Within each turn, it is updated autoregressively, incorporating each position only after computing its anchor representation to preserve causality. This avoids persistent cross-turn state storage but requires recomputing historical representations at each turn.

Thanks again for your careful and valuable comments. We hope our response addresses your concerns. If our responses answer your questions well and reassure your concerns, we would appreciate it if you could reassess our work and increase the rating.

Reviewer_h7saOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpeechAnchor, a Gated DeltaNet-based module for improving semantic, paralinguistic, and speaker-style consistency in multi-turn spoken dialogue. Rather than relying solely on the LLM backbone to model dialogue history, it provides an additional history-modeling pathway for the speech decoder. Combined with a carefully designed synthetic data pipeline, the framework demonstrates improvements across several benchmarks.

Critical strengths
  • The paper addresses an important problem. Modeling long speech histories places a substantial burden on SLMs, and the dedicated module offers a sensible approach to improving cross-turn consistency.
  • Improvements on two different SLM backbones suggest that the approach is applicable beyond a single model.
  • The evaluation uses multiple existing benchmarks rather than relying solely on a benchmark constructed through the proposed data pipeline. This strengthens the empirical evidence for the approach.
Critical weaknesses

My main concerns relate to the validation of the synthesized training data and the model’s generated responses, rather than the overall methodology.

  • The reported quality checks for the synthesized data focus on UTMOS and speaker-embedding similarity. These provide useful evidence of naturalness and speaker identity consistency, but do not directly establish whether the speech accurately conveys the intended text or realizes the annotated paralinguistic cues. Listening evaluations with multiple raters would help validate the data.
  • The evaluation of model outputs would similarly benefit from human assessment. Although the benchmark results and speaker-similarity analysis are encouraging, it remains unclear whether the improvements translate into perceptually better paralinguistic appropriateness and cross-turn speaker-style consistency. Speaker identity similarity alone does not fully capture style attributes such as speaking rate and volume.
Questions for authors
  • How was the synthesized speech checked for transcription fidelity and pronunciation errors, including omissions, repetitions, and mispronunciations? Were ASR-based filtering or manual checks performed?
  • Could the authors clarify whether using outputs from Gemini-2.5-Pro-Preview-TTS to train the proposed models is permitted under the applicable terms? Given the restrictions on competing-model development, please specify the service used and any relevant permission or applicable provision.
  • Could the authors provide listening evaluations of both the synthesized data and the model outputs with multiple independent raters, reporting the number of raters and how they were recruited? For the data, it would be useful to assess whether the intended paralinguistic cues are perceptible and contextually appropriate. For model outputs, a blind comparison against the backbones would help establish whether the improvements in paralinguistic appropriateness and speaker-style consistency are perceptually meaningful.

I would be willing to raise my score if additional clarification and evaluation adequately address these concerns.

7 comments

Q1: About the quality validation of synthesized speech data.

A1: Thank the reviewer for this suggestion. We conducted four complementary evaluations:

  • Validation of Transcription Fidelity in Synthesized Speech

    We performed full-corpus ASR evaluation on all 100,000 synthesized instances, covering both user and assistant speech. We used Whisper-Large-v3 to transcribe the synthesized audio, taking the actual text fed into the TTS model as the reference. After normalizing capitalization, punctuation, and numerical formats, we calculated Word Error Rate (WER) and substitution, deletion, and insertion rates.

    DatasetWER ↓Substitution ↓Deletion ↓Insertion ↓
    Semantic3.0%1.0%1.3%0.7%
    Paralinguistic4.0%1.4%1.7%0.9%
    Overall3.2%1.1%1.4%0.7%

    Overall, the synthesized speech achieves a low WER of 3.2%, indicating high transcription fidelity across both datasets.

  • Validation of Pronunciation Accuracy in Synthesized Speech

    To complement ASR evaluation with direct listening assessments, we randomly sampled 100 turns (80 Semantic and 20 Paralinguistic), comprising 200 synthesized utterances. Each utterance was independently evaluated by three raters.The raters were Ph.D. students specializing in LLM research, recruited from the authors' university laboratory, and all had near-native proficiency in English. They independently listened to each audio using headphones in a quiet environment and compared it against the corresponding TTS input text, assessing three aspects using binary Yes/No questions: (1) omission of words or content, (2) unintended repetition, and (3) mispronunciation. When identifying an error, raters marked the corresponding word or segment. Acceptable accent variations and natural connected or reduced speech were not considered pronunciation errors. At least two of the three raters had to agree for an error to be counted.

    Evaluation MetricResult ↓
    Omission Rate1.5%
    Repetition Rate1.0%
    Mispronunciation Rate2.5%

    The inter-rater agreement, measured by nominal Krippendorff's Alpha, was 0.73. These results indicate that omissions, unintended repetitions, and pronunciation errors are infrequent in the synthesized speech.

  • Validation of Contextual Paralinguistic Consistency

    We randomly sampled 600 turns without replacement from the Paralinguistic Consistency Data. The same three raters independently evaluated each turn (1,800 judgments in total), based on the preceding dialogue history (texts, paralinguistic annotations, and synthesized speech), the current user utterance and its annotation, and the assistant's textual response. Using a 5-point Likert scale (1: Completely inappropriate; 5: Highly appropriate), they assessed (1) the contextual appropriateness of the user's paralinguistic changes across turns and (2) the appropriateness of the assistant's response to the user's intent and paralinguistic information in context.

    Evaluation DimensionMean Rating ↑
    Paralinguistic Change Appropriateness4.25
    Assistant Response Appropriateness4.30

    The ordinal Krippendorff's alpha was 0.73. Together, these results support the effectiveness of our data construction pipeline in promoting contextual paralinguistic consistency.

  • Validation of Paralinguistic Cue Realization

    We conducted an additional listening evaluation on 600 different turns from the Paralinguistic Consistency Data. The evaluation focused on the synthesized user-side speech corresponding to the annotated paralinguistic cues. Raters were provided with the current user utterance, its original paralinguistic annotation, and the corresponding synthesized speech. Using a 5-point Likert scale (1: Not realized at all; 5: Fully realized), they assessed the extent to which the synthesized speech realized the annotated paralinguistic information.

    Evaluation DimensionMean Rating ↑
    Paralinguistic Cue Realization4.15

    The ordinal Krippendorff's alpha was 0.72. These results support the effectiveness of our TTS synthesis pipeline in realizing the intended paralinguistic cues.

These evaluations provide complementary evidence of the synthesized data's transcription fidelity, pronunciation accuracy, contextual paralinguistic consistency, and cue realization. We will incorporate the corresponding protocols and results into the revised manuscript.

Q2: About human evaluation of model output speech.

A2: We thank the reviewer for this valuable suggestion. We conducted two additional human evaluations using blind pairwise comparisons against the corresponding backbone models. All evaluations were conducted by the same three independent raters and followed the rater recruitment, listening environment, and assessment protocol described in our response to Q1. All comparisons were blinded to model identities, with randomized A/B assignments and presentation orders.

  • Human Evaluation of Paralinguistic Appropriateness.

    We randomly sampled 50 test samples from the Paralinguistic Track of MTalk-Bench. For each sample, we used the original evaluation rubric provided by MTalk-Bench. These rubrics provide detailed and well-defined criteria for assessing paralinguistic capabilities. Given identical user speech inputs and dialogue contexts, we generated responses using each backbone and its corresponding trained SpeechAnchor model, resulting in 100 blind pairwise comparisons across the MiMo and Kimi model groups.

    Raters were provided with the dialogue context, current user speech, the original MTalk-Bench evaluation rubric, and two candidate speech responses. They were asked to perform an overall rubric-based comparison, determining which response better satisfied the sample-specific evaluation criteria. The evaluation used a single question, Rubric-based Preference, with three response options: A, B, or No noticeable difference. Model identities were concealed, and A/B assignments and presentation orders were randomized.

    Model ComparisonEvaluation DimensionSpeechAnchor Preference ↑Backbone Preference ↓Tie
    MiMoRubric-based Preference62.0%21.0%17.0%
    KimiRubric-based Preference59.0%23.0%18.0%

    The nominal Krippendorff's Alpha was 0.68. These results show that SpeechAnchor is consistently preferred over its corresponding backbone models, demonstrating its effectiveness in improving the paralinguistic appropriateness of generated speech responses.

  • Human Evaluation of Cross-turn Speaker-Style Consistency.

    We randomly sampled 50 complete dialogues from a unified pool of the three multi-turn benchmarks used in our paper. From each generated dialogue, we extracted only the first and last assistant speech responses to assess cross-turn speaker-style consistency. This resulted in 100 blind pairwise comparisons across the MiMo and Kimi model groups.

    Raters were provided with the dialogue context and the first and last assistant responses from each system. They compared the consistency between the two responses across three dimensions: (1) Speaking Rate Consistency, assessing speaking rate stability; (2) Volume Consistency, assessing perceived loudness stability; and (3) Overall Speaker-Style Consistency, considering vocal timbre, pitch, speaking rate, volume, and habitual prosodic characteristics.

    The evaluation also adopted blind pairwise comparison: model identities were concealed, A/B labels and presentation orders were randomized, and the first and last responses within each system were presented in their original chronological order. No independent loudness normalization was applied to individual utterances, and both systems' audio were presented using the same playback device and fixed volume. Reasonable variations in speaking style required by the conversational context were not considered inconsistencies.

    Model ComparisonEvaluation DimensionSpeechAnchor Preference ↑Backbone Preference ↓Tie
    MiMoSpeaking Rate Consistency63.5%21.0%15.5%
    MiMoVolume Consistency60.0%23.5%16.5%
    MiMoOverall Speaker-Style Consistency66.0%18.5%15.5%
    KimiSpeaking Rate Consistency58.5%25.0%16.5%
    KimiVolume Consistency54.5%28.0%17.5%
    KimiOverall Speaker-Style Consistency61.5%23.0%15.5%

    The average nominal Krippendorff's Alpha across the four evaluation questions was 0.70. This confirms that the improvements achieved by SpeechAnchor translate into perceptually meaningful gains in cross-turn speaker-style consistency.

Overall, these blind listening evaluations provide complementary human evidence that SpeechAnchor improves both paralinguistic appropriateness and cross-turn speaker-style consistency across different backbone models.

Q3: About transcription fidelity and pronunciation error filtering.

A3: We thank the reviewer for raising this important question. As detailed in our response to Q1, we conducted full-corpus ASR evaluation using Whisper-Large-v3 on all 100,000 synthesized speech instances, achieving an overall WER of 3.2%. We further conducted manual listening evaluations to assess omissions, unintended repetitions, and mispronunciations, with corresponding error rates of 1.5%, 1.0%, and 2.5%, respectively. These complementary evaluations provide direct evidence of the transcription fidelity and pronunciation accuracy of our synthesized data.

Q4: About the licensing and permitted use of Gemini TTS outputs.

A4: We appreciate the reviewer for bringing this important issue to our attention.

We clarify that we used Google's Gemini Developer API with the model gemini-2.5-pro-preview-tts to synthesize speech samples from text-based dialogues and paralinguistic style prompts. The generated audio samples were subsequently used as training data for our proposed SpeechAnchor models.

We acknowledge the reviewer's concern regarding the applicable terms of service and restrictions on developing competing models. To ensure compliance, we will contact Google through the account used to access the Gemini Developer API to seek explicit clarification and obtain the necessary authorization for this use.

Q5: About multi-rater listening evaluations.

A5: Thank the reviewer for this valuable suggestion. As detailed in our responses to Q1 and Q2, we conducted additional human evaluations using the same three independent raters, whose recruitment criteria and evaluation protocols are described in Q1. These evaluations cover both synthesized data quality and model-generated speech, including transcription fidelity, pronunciation accuracy, contextual paralinguistic consistency, paralinguistic cue realization, paralinguistic appropriateness, and cross-turn speaker-style consistency.

Thanks again for your careful and valuable comments. We hope our response addresses your concerns. If our responses answer your questions well and reassure your concerns, we would appreciate it if you could reassess our work and increase the rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.