acceptodds
Under review as a conference paper at ICLR 2027

Silent Tags: Expressive Speech without Language-Backbone Updates

Abstract

With the rapid development of omni large language models, users can converse directly with assistants through speech. However, most existing models render every response in the same flat prosody, regardless of whether the user is distressed, excited, or amused, limiting the user experience in human-machine interaction. To inject expressiveness, the prevailing approach trains the language backbone, namely the Thinker, to jointly predict paralinguistic tags alongside text. We find that this consistently degrades reasoning on benchmarks such as MMLU and VoiceBench. A key challenge therefore lies in how to make a speech assistant expressive without compromising the reasoning ability of its language backbone. To address this challenge, we propose Silent-Tags, a Thinker-agnostic framework that leaves the Thinker entirely frozen and trains only the Talker. Our key insight is that a well-trained omni Thinker already encodes, in its hidden states, contextual signals required for emotional and paralinguistic inference. Silent-Tags equips the Talker with lightweight cross-attention layers that selectively tap these hidden states for rich emotional and paralinguistic cues, supervised by a tag-controllable TTS teacher together with a segment-level auxiliary classifier. Silent-Tags introduces no extra Thinker decoding steps and no tag-token generation at inference time; the only added overhead is the Talker's cross-attention into already-cached Thinker states. On ParaS2SBench and EmergentTTS-Eval, Silent-Tags consistently outperforms explicit-paradigm baselines on expressiveness. Audio samples, code, and the full tag vocabulary are available at [link](https://anonymous-3911.pages.dev/).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Paralinguistic control, Omni speech

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LTkBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Silent Tags, a framework for improving the paralinguistic expressiveness of speech-to-speech (S2S) models while preserving the reasoning capabilities of the underlying language model. The system freezes the language model's Thinker and trains only the Talker. Lightweight cross-attention layers allow the Talker to extract response-conditioned paralinguistic information from the Thinker's hidden states, enabling expressive speech generation without explicitly generating paralinguistic tag tokens at inference time. The authors evaluate the system's expressiveness across multiple benchmarks. The results show competitive expressiveness while preserving the frozen Thinker's text-side performance.

Critical strengths

The proposed method is simple yet effective. The authors also conduct an extensive set of experiments across multiple benchmarks and controlled settings. The experiments provide convincing evidence that the proposed method can enhance speech expressiveness while preserving the reasoning capabilities of the model.

Critical weaknesses
  1. The paper suffers from severe writing and presentation issues, including excessive and obscure terminology, inconsistent mathematical notation, and undefined or unused symbols. The figures are unclear. For example, models mentioned in the captions, such as Fish Audio and CosyVoice 2, are not clearly identifiable in the figures themselves, making it difficult to understand where or how they are used. And the experimental setup lacks sufficient details about baseline configurations. In addition, inconsistent evaluation protocols across tables undermine the comparability and reliability of the reported results. For example, the ablation study table in the appendix explicitly states, “These results are not directly comparable to Table 1,” which raises concerns about the consistency and comparability of the reported results.
  2. The core idea of leveraging intermediate hidden states from a frozen language model for speech generation, while preserving its existing reasoning capabilities is not novelty enough. May prior works explored this direction such like: @inproceedings{hong2026freezeempath, title={FreezeEmpath: Efficient Training for Empathetic Spoken Chatbots with Frozen LLMs}, author={Hong, Yun and Zhou, Yan and Feng, Yang}, booktitle={Findings of the Association for Computational Linguistics: ACL 2026}, pages={17141--17157}, year={2026} }
Questions for authors

Fig. 2 probes only three non-verbal events (Laughter, Sigh, Hesitation) on Qwen2.5-Omni, whereas the readout must support 48 categories, most of which are emotions and speaking styles. The text acknowledges that the probe "neither establishes coverage of all 48 tags nor isolates acoustic cues from correlated responsetext information." Could the authors provide further evidence that the information in layers 0–6 is sufficient to support expressive speech generation across the full tag set?

Reviewer_MpHmOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Silent Tags, a method to improve expressive speech generation without updating the Thinker. The method introduces cross-attention layers in the Talker to access the Thinker's lower-layer hidden states, trained with synthetic expressive speech and segment-level supervision.

Critical strengths
  1. Important and timely research direction. As speech-to-speech interaction becomes increasingly important, improving contextually appropriate expressive speech is a meaningful research direction.

  2. Simple and effective approach. The proposed method is relatively simple and achieves promising results without updating the Thinker or requiring explicit tag generation.

Critical weaknesses
  1. Implementation details and reproducibility. Several important implementation details are unclear:
  • How is the original top-layer Thinker conditioning integrated with the newly introduced cross-attention?
  • How are the seven layer-specific cross-attention outputs aggregated into the pooled silent tag?
  • What exactly are the prediction targets in the layer-wise probing experiment?
  • Based on the released code, is CosyVoice used as the Talker for both Qwen and Kimi? How does this correspond to the architecture described in the paper?
  • The Talker model/checkpoints do not appear to be released. While this is understandable, the limited implementation details and apparent discrepancies between the paper and code might make reproducibility difficult.
  1. [Minor] Writing and presentation could be improved. I found some parts of the paper difficult to follow. For example, in Lines 62–69 of the Introduction, are not easy to follow. The discussion suddenly introduces teacher-generated data without clearly explaining the role of the teacher or how it relates to the proposed method.
Questions for authors

None

Reviewer_2a9YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies expressive speech generation without updating the language backbone of an omni model. Silent Tags adds cross-attention to the Talker, reading cached response representations from the bottom seven Thinker layers (layers 0-6). Training uses approximately 90k synthesized expressive dialogue pairs with segment-level tag classification and style-embedding distillation in addition to codec prediction. At inference, the Thinker first generates the full text response and the Talker then renders speech without explicit tag generation. The evaluation includes a reconstructed ParaS2SBench, EmergentTTS-Eval, text-side reasoning benchmarks, a fixed-source/varying-response diagnostic, and a blinded listening study. The results support competitive expressive rendering with unchanged Thinker behavior, although they do not establish uniform superiority over explicit-tag methods.

Critical strengths

The work addresses a meaningful separation between linguistic competence and expressive rendering. Freezing the Thinker gives a clear preservation property for its text-side behavior. Importantly, the paper compares against Talker fine-tuning on the same expressive data rather than attributing every gain to the new architecture.

The paper also provides thorough investigation of layer selection, including layer-wise probing of paralinguistic events and comparisons between lower-layer and upper-layer conditioning. These analyses offer useful empirical motivation for the selected readout layers and help explain the design.

Critical weaknesses

The main weakness of the paper lies in the dependence on synthetic supervision and unclear advantages over a strong LLM+prompted-TTS pipeline.

Expressiveness is constrained by the prompted-TTS supervision paradigm. It primarily learns to reproduce the explicit prompting pipeline through implicit conditioning. Silent Tags also learns expressive rendering from teacher-generated speech and style targets, which may constrain the diversity and fidelity of the behaviors it acquires. Changing the synthesis teacher does not address this underlying limitation. It therefore remains unclear whether the method learns richer, context-sensitive delivery beyond this construction paradigm.

Moreover, while the paper evaluates Qwen2.5-Omni, Kimi-Audio, and Qwen3-Omni, teacher sensitivity is studied primarily on Qwen2.5-Omni, leaving generalization across backbone teacher combinations less established. A matched comparison using a newer controllable synthesizer, such as Fun-CosyVoice 3.0, would help clarify the method’s practical advantage beyond distilling the teacher’s expressive capabilities.

Questions for authors
  1. Since Sec 3.3 generates the complete response before speech decoding, what are time-to-first-audio and total inference cost relative to the same-data and explicit-tag baselines?
  2. How to ensure smooth and natural transition in emotion between sentences in the data generation pipeline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.