- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes X-EMO, a cross-lingual zero-shot expressive TTS framework that disentangles linguistic content from speaker identity and emotion through conditional speech tokenization and an information bottleneck. It combines an autoregressive language model with a flow-matching acoustic model to generate expressive speech across languages. Experiments on four languages demonstrate competitive performance in intelligibility, speaker similarity, and emotion preservation.
Critical strengths
- The use of an attribute-matched conditioning utterance provides a reasonable way to encourage linguistic and paralinguistic information to be represented separately. Compared with explicitly factorizing speech into predefined components, this design is relatively simple and can be integrated into existing token-based TTS architectures.
- The system achieves competitive intelligibility and emotion preservation across four languages.
Critical weaknesses
-
The central contribution is a tokenizer that removes speaker and emotion information while preserving linguistic content. However, the experiments mainly evaluate downstream TTS performance rather than the actual information retained in the discrete tokens. The improvements in Tables 1 and 2 do not necessarily demonstrate successful disentanglement, since they could also result from changes in token predictability or reconstruction quality. More importantly, Table 3 shows that adding expressive data decreases all five emotion preservation metrics in the zh→en direction, despite substantially improving WER. This raises questions about whether the proposed tokenizer consistently separates emotion from linguistic content.
-
The cross-lingual inference procedure introduces a fixed native-speaker anchor for the target language, while same-language synthesis uses the original prompt as the anchor. These different prompting strategies make it difficult to determine how much of the reported cross-lingual robustness comes from the proposed tokenizer. This concern is particularly relevant to Table 5, where the system achieves substantially lower WER in several cross-lingual settings than in the corresponding same-language setting. For example, Indonesian synthesis obtains WERs around 3.1 with cross-lingual prompts, compared with 5.83 with an Indonesian prompt. The authors also acknowledge that anchor quality substantially affects linguistic accuracy. Therefore, the current results do not clearly isolate the benefit of the proposed disentanglement mechanism from that of using a curated target-language anchor.
Questions for authors
Q1. Can the authors provide direct evidence that the proposed tokenizer removes speaker and emotion information from the discrete tokens? For example, comparing speaker and emotion classification performance using tokens learned with and without the proposed conditioning mechanism would help substantiate the central disentanglement claim. How do the authors interpret the consistent degradation in zh→en emotion preservation shown in Table 3?
Q2. How much of the cross-lingual improvement comes from the target-language anchor rather than the disentangled tokenizer? In particular, can the authors compare the proposed tokenizer with a conventional tokenizer under the same fixed-anchor inference setting? This would help determine whether the observed cross-lingual robustness is primarily attributable to the proposed representation.
3 comments
Question 1: Can the authors provide direct evidence that the proposed tokenizer removes speaker and emotion information from the discrete tokens? For example, comparing speaker and emotion classification performance using tokens learned with and without the proposed conditioning mechanism would help substantiate the central disentanglement claim. How do the authors interpret the consistent degradation in zh→en emotion preservation shown in Table 3?
**Response: ** We thank the reviewer for these questions. Please refer to our responses above for the discussion of the zh to en emotion preservation results and additional probing experiments in Section 5.1.
Question 2: How much of the cross-lingual improvement comes from the target-language anchor rather than the disentangled tokenizer? In particular, can the authors compare the proposed tokenizer with a conventional tokenizer under the same fixed-anchor inference setting? This would help determine whether the observed cross-lingual robustness is primarily attributable to the proposed representation.
**Response: ** We thank the reviewer for this question. This concern is closely related to Weakness 2. Please refer to our response above.
Thank you for your detailed response and for the additional analyses and comparisons. I appreciate your careful consideration of the concerns raised, your acknowledgment of the limitations, and the effort you have made to clarify the results in the revised manuscript.
Weakness 1: the experiments mainly evaluate downstream TTS performance rather than the actual information retained in the discrete tokens. Table 3 shows that adding expressive data decreases all five emotion preservation metrics in the zh→en direction, despite substantially improving WER. This raises questions about whether the proposed tokenizer consistently separates emotion from linguistic content.
Response: We thank the reviewer for pointing out this important issue. We agree that downstream TTS metrics alone do not provide direct evidence of what information is retained in the discrete tokens, and that the degradation in emotion preservation for zh→en warrants further investigation.
In the revised paper, we have added a linear probing study to directly examine speaker and emotion information retained in the discrete tokens (please see our common response ``Direct Evidence for Disentanglement"). The results show that our proposed decoder conditioning reduces both speaker and emotion information compared with the tokenizer trained with the information bottleneck alone. These results provide direct evidence that the proposed tokenizer reduces speaker and emotion information, although the disentanglement is not complete, particularly for emotion.
We acknowledge that all five emotion preservation metrics decrease in the zh→en direction. We hypothesis that this is because emotion information from the prompt must be conveyed through the flow-matching model, which relies on an utterance-level Emotion2Vec embedding and an acoustic prefix extracted from the prompt mel-spectrogram. Unlike English, Chinese is a tonal language, where pitch contours encode both lexical tones and emotional intonation. These two types of pitch variation may be difficult to distinguish using a single utterance-level emotion embedding and a short acoustic prefix. The zh→en results further highlight that effective emotion preservation depends not only on token disentanglement but also on the capacity of the acoustic conditioning mechanism.
Weakness 2: The cross-lingual inference procedure introduces a fixed native-speaker anchor for the target language, while same-language synthesis uses the original prompt as the anchor. These different prompting strategies make it difficult to determine how much of the reported cross-lingual robustness comes from the proposed tokenizer. The current results do not clearly isolate the benefit of the proposed disentanglement mechanism from that of using a curated target-language anchor.
Response: We thank the reviewer for pointing out that the contribution of the proposed tokenizer is confounded with that of the fixed target-language anchor in our original experiments. We agree that these two contributions should be better separated. In the revised manuscript, we have added two controlled comparisons that keep the anchor and inference procedure unchanged while varying the tokenizer.
We further replace our tokenizer with two officially released tokenizers: the content tokenizer of Vevo and the supervised S3 tokenizer of CosyVoice. We keep the downstream architecture, training data, and fixed-anchor inference procedure unchanged. The results are added in Table 8 of Appendix A.6 of the revised paper. These results shown here highlight the complementary roles of the anchor and the tokenizer. The fixed anchor itself contributes to cross-lingual intelligibility, as conventional tokenizers also achieve reasonable results under this setting. However, our tokenizer achieves the highest speaker similarity in both cross-lingual directions, outperforming S3 and Vevo. Moreover, our tokenizer achieves the highest MOS in all four directions, suggesting that its benefits extend beyond speaker similarity to overall speech quality.
Table: Comparison of speech tokenizers under the same fixed-anchor inference setting. In the same-language directions, the prompt utterance serves as the anchor. Bold indicates the best result.