acceptodds
Under review as a conference paper at ICLR 2027

X-Emo: Disentangled Speech Tokens for Cross-Lingual Zero-Shot Expressive Text-to-Speech

Abstract

Cross-lingual zero-shot expressive text-to-speech (TTS) aims to generate speech in a target language while preserving the speaker identity and emotion of a prompt utterance in another language. A key challenge is to transfer these attributes without entangling them with linguistic content and source-language characteristics. We propose X-EMO, a cross-lingual expressive TTS framework built on a disentangled discrete speech tokenizer. During tokenizer training, the decoder receives an additional utterance that shares the speaker and emotion of the input but contains different linguistic content. This makes speaker and emotional information directly available to the decoder, encouraging the discrete representation to primarily encode information required for linguistic reconstruction. The resulting tokens are modeled by an autoregressive language model, while a flow-matching acoustic model synthesizes speech conditioned on speaker and emotion embeddings from the prompt utterance. With approximately 65 hours of expressive speech added to bilingual read-speech data, X-EMO achieves competitive intelligibility, speaker similarity, naturalness, and emotion preservation across multiple languages, while maintaining stable cross-lingual performance across diverse prompt languages.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Cross-lingual Text-to-Speech, Expressive Speech Synthesis, Speech Disentanglement, Speech Tokenization, Zero-shot Text-to-Speech

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZZpTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes X-EMO, a cross-lingual zero-shot expressive TTS framework that disentangles linguistic content from speaker identity and emotion through conditional speech tokenization and an information bottleneck. It combines an autoregressive language model with a flow-matching acoustic model to generate expressive speech across languages. Experiments on four languages demonstrate competitive performance in intelligibility, speaker similarity, and emotion preservation.

Critical strengths
  1. The use of an attribute-matched conditioning utterance provides a reasonable way to encourage linguistic and paralinguistic information to be represented separately. Compared with explicitly factorizing speech into predefined components, this design is relatively simple and can be integrated into existing token-based TTS architectures.
  2. The system achieves competitive intelligibility and emotion preservation across four languages.
Critical weaknesses
  1. The central contribution is a tokenizer that removes speaker and emotion information while preserving linguistic content. However, the experiments mainly evaluate downstream TTS performance rather than the actual information retained in the discrete tokens. The improvements in Tables 1 and 2 do not necessarily demonstrate successful disentanglement, since they could also result from changes in token predictability or reconstruction quality. More importantly, Table 3 shows that adding expressive data decreases all five emotion preservation metrics in the zh→en direction, despite substantially improving WER. This raises questions about whether the proposed tokenizer consistently separates emotion from linguistic content.

  2. The cross-lingual inference procedure introduces a fixed native-speaker anchor for the target language, while same-language synthesis uses the original prompt as the anchor. These different prompting strategies make it difficult to determine how much of the reported cross-lingual robustness comes from the proposed tokenizer. This concern is particularly relevant to Table 5, where the system achieves substantially lower WER in several cross-lingual settings than in the corresponding same-language setting. For example, Indonesian synthesis obtains WERs around 3.1 with cross-lingual prompts, compared with 5.83 with an Indonesian prompt. The authors also acknowledge that anchor quality substantially affects linguistic accuracy. Therefore, the current results do not clearly isolate the benefit of the proposed disentanglement mechanism from that of using a curated target-language anchor.

Questions for authors

Q1. Can the authors provide direct evidence that the proposed tokenizer removes speaker and emotion information from the discrete tokens? For example, comparing speaker and emotion classification performance using tokens learned with and without the proposed conditioning mechanism would help substantiate the central disentanglement claim. How do the authors interpret the consistent degradation in zh→en emotion preservation shown in Table 3?

Q2. How much of the cross-lingual improvement comes from the target-language anchor rather than the disentangled tokenizer? In particular, can the authors compare the proposed tokenizer with a conventional tokenizer under the same fixed-anchor inference setting? This would help determine whether the observed cross-lingual robustness is primarily attributable to the proposed representation.

3 comments

Weakness 1: the experiments mainly evaluate downstream TTS performance rather than the actual information retained in the discrete tokens. Table 3 shows that adding expressive data decreases all five emotion preservation metrics in the zh→en direction, despite substantially improving WER. This raises questions about whether the proposed tokenizer consistently separates emotion from linguistic content.

Response: We thank the reviewer for pointing out this important issue. We agree that downstream TTS metrics alone do not provide direct evidence of what information is retained in the discrete tokens, and that the degradation in emotion preservation for zh→en warrants further investigation.

In the revised paper, we have added a linear probing study to directly examine speaker and emotion information retained in the discrete tokens (please see our common response ``Direct Evidence for Disentanglement"). The results show that our proposed decoder conditioning reduces both speaker and emotion information compared with the tokenizer trained with the information bottleneck alone. These results provide direct evidence that the proposed tokenizer reduces speaker and emotion information, although the disentanglement is not complete, particularly for emotion.

We acknowledge that all five emotion preservation metrics decrease in the zh→en direction. We hypothesis that this is because emotion information from the prompt must be conveyed through the flow-matching model, which relies on an utterance-level Emotion2Vec embedding and an acoustic prefix extracted from the prompt mel-spectrogram. Unlike English, Chinese is a tonal language, where pitch contours encode both lexical tones and emotional intonation. These two types of pitch variation may be difficult to distinguish using a single utterance-level emotion embedding and a short acoustic prefix. The zh→en results further highlight that effective emotion preservation depends not only on token disentanglement but also on the capacity of the acoustic conditioning mechanism.

Weakness 2: The cross-lingual inference procedure introduces a fixed native-speaker anchor for the target language, while same-language synthesis uses the original prompt as the anchor. These different prompting strategies make it difficult to determine how much of the reported cross-lingual robustness comes from the proposed tokenizer. The current results do not clearly isolate the benefit of the proposed disentanglement mechanism from that of using a curated target-language anchor.

Response: We thank the reviewer for pointing out that the contribution of the proposed tokenizer is confounded with that of the fixed target-language anchor in our original experiments. We agree that these two contributions should be better separated. In the revised manuscript, we have added two controlled comparisons that keep the anchor and inference procedure unchanged while varying the tokenizer.

We further replace our tokenizer with two officially released tokenizers: the content tokenizer of Vevo and the supervised S3 tokenizer of CosyVoice. We keep the downstream architecture, training data, and fixed-anchor inference procedure unchanged. The results are added in Table 8 of Appendix A.6 of the revised paper. These results shown here highlight the complementary roles of the anchor and the tokenizer. The fixed anchor itself contributes to cross-lingual intelligibility, as conventional tokenizers also achieve reasonable results under this setting. However, our tokenizer achieves the highest speaker similarity in both cross-lingual directions, outperforming S3 and Vevo. Moreover, our tokenizer achieves the highest MOS in all four directions, suggesting that its benefits extend beyond speaker similarity to overall speech quality.

Table: Comparison of speech tokenizers under the same fixed-anchor inference setting. In the same-language directions, the prompt utterance serves as the anchor. Bold indicates the best result.

DirectionMetricVevoCosyVoice S3Ours
en→enWER ↓12.468.997.03
en→enSIM ↑0.5700.5380.567
en→enMOS ↑2.862.832.89
zh→zhCER ↓12.844.824.92
zh→zhSIM ↑0.6960.6850.705
zh→zhMOS ↑2.642.642.71
en→zhCER ↓13.324.946.27
en→zhSIM ↑0.3710.3260.405
en→zhMOS ↑2.592.612.65
zh→enWER ↓11.288.896.40
zh→enSIM ↑0.4360.4120.479
zh→enMOS ↑2.982.912.99
We also compare our tokenizer with a No-prompt variant, which shares the same architecture, codebook, information bottleneck configuration, and training data. The detail can be found in common respond.

Question 1: Can the authors provide direct evidence that the proposed tokenizer removes speaker and emotion information from the discrete tokens? For example, comparing speaker and emotion classification performance using tokens learned with and without the proposed conditioning mechanism would help substantiate the central disentanglement claim. How do the authors interpret the consistent degradation in zh→en emotion preservation shown in Table 3?

**Response: ** We thank the reviewer for these questions. Please refer to our responses above for the discussion of the zh to en emotion preservation results and additional probing experiments in Section 5.1.

Question 2: How much of the cross-lingual improvement comes from the target-language anchor rather than the disentangled tokenizer? In particular, can the authors compare the proposed tokenizer with a conventional tokenizer under the same fixed-anchor inference setting? This would help determine whether the observed cross-lingual robustness is primarily attributable to the proposed representation.

**Response: ** We thank the reviewer for this question. This concern is closely related to Weakness 2. Please refer to our response above.

Thank you for your detailed response and for the additional analyses and comparisons. I appreciate your careful consideration of the concerns raised, your acknowledgment of the limitations, and the effort you have made to clarify the results in the revised manuscript.

Reviewer_dkdwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes X-EMO, a cross-lingual zero-shot expressive TTS framework designed to disentangle linguistic content from speaker identity and emotion in discrete speech tokens. During tokenizer training, the decoder is conditioned on a separate utterance with the same speaker and emotion but different linguistic content. A KL-based information cost encourages the tokens to retain linguistic information while reducing redundant speaker and emotion information.

Critical strengths

S1. Distinctive Approach to Disentangled Speech Tokenization

The paper introduces a distinctive approach to disentangled speech tokenization by conditioning the tokenizer decoder on a separate utterance with the same speaker and emotion but different linguistic content. This provides an interesting perspective on learning disentangled speech representations for cross-lingual expressive TTS.

S2. Promising Cross-Lingual Performance and Comprehensive Evaluation

The paper provides extensive evaluations across four languages, including ablation studies on information cost and expressive training data.

Critical weaknesses

W1. Limited Direct Validation of Disentanglement and Emotion Preservation

Although the improvements in WER/CER provide meaningful indirect evidence supporting the proposed disentanglement strategy, they may not fully demonstrate the extent to which speaker and emotion information is separated from linguistic content in the discrete speech tokens. Furthermore, emotion preservation is evaluated solely through Emotion2Vec-based classification, despite Emotion2Vec also being used for emotion conditioning. No subjective listening tests are reported, and the paper does not provide a demo page or audio samples for qualitative assessment. These limitations weaken the experimental evidence supporting the paper's main contributions.

W2. Limited Evidence for the Claimed Data-Efficiency Advantage

The paper emphasizes that the proposed disentangled speech tokenizer is trained with approximately 65 hours of expressive speech in addition to bilingual read-speech data. However, using a relatively small amount of expressive data for tokenizer training alone does not necessarily establish a distinctive advantage in data efficiency. Without comparisons against alternative approaches under comparable expressive-data budgets, it remains unclear whether the proposed method offers a meaningful advantage in data efficiency. This limits the evidence supporting data efficiency as a key contribution of the paper.

Questions for authors

Q1. Evidence of Disentanglement

Does the reduction in WER/CER necessarily indicate that speaker and emotion information has been effectively removed from the discrete speech tokens? Could these improvements instead result from the information bottleneck reducing token complexity or producing a distribution that is easier for the language model to predict?

Q2. Relationship Between Disentanglement and Speaker Similarity

In Table 2, the average WER/CER decreases substantially from 8.00 to 6.16, while speaker similarity (SIM) increases only marginally from 0.529 to 0.539. If disentanglement reduces speaker and emotion information that may overlap with the separate conditioning embeddings in the flow-matching model, might a more noticeable improvement in SIM be expected? Could the authors explain why linguistic accuracy improves substantially while SIM remains nearly unchanged?

Q3. Data Efficiency of the Proposed Tokenizer

The paper emphasizes using approximately 65 hours of expressive speech during disentangled tokenizer training. Could the authors clarify whether the proposed approach offers a data-efficiency advantage over conventional speech tokenizers or alternative disentanglement approaches under comparable expressive-data budgets? Such evidence would help assess whether the reported data efficiency is a distinctive benefit of the proposed method.

6 comments

Weakness 1: Limited Direct Validation of Disentanglement and Emotion Preservation. Emotion preservation is evaluated solely through Emotion2Vec-based classification. No subjective listening tests are reported, and the paper does not provide a demo page or audio samples for qualitative assessment

Response: We thank the reviewer for raising these concerns. We have addressed them in our common response. Please refer to “Independent Emotion Evaluation and Audio Samples” for the evaluation using an independent emotion classifier and the released audio demo. These additions provide complementary quantitative and qualitative evidence for disentanglement and emotion preservation.

Weakness 2: Limited Evidence for the Claimed Data-Efficiency Advantage. The paper emphasizes that the proposed disentangled speech tokenizer is trained with approximately 65 hours of expressive speech in addition to bilingual read-speech data. However, using a relatively small amount of expressive data for tokenizer training alone does not necessarily establish a distinctive advantage in data efficiency. Without comparisons against alternative approaches under comparable expressive-data budgets, it remains unclear whether the proposed method offers a meaningful advantage in data efficiency. This limits the evidence supporting data efficiency as a key contribution of the paper.

Response: We thank the reviewer for pointing out this ambiguity. We agree that using approximately 65 hours of expressive speech for tokenizer training alone does not establish a data-efficiency advantage over other methods, particularly without a comparison under matched expressive-data budgets.

Our intended data-efficiency claim concerns the overall speech training data required by the TTS system, rather than the amount of expressive data used to train the tokenizer. Specifically, as reported in Table 6 of the revised paper, X-EMO is trained with approximately 100K hours of speech, compared with approximately 500K hours for Confucius4-TTS, 581K hours for OmniVoice, 1M hours for MOSS-TTS, and over 2M hours for VoxCPM2. Despite this substantially smaller reported training-data budget, X-EMO achieves competitive performance across the evaluated languages, including low-resource Southeast Asian languages. We acknowledge that these cross-system comparisons are not strictly controlled comparisons under matched training settings, and therefore do not establish a definitive causal relationship between data scale and performance.

To avoid conflating the two aspects, we have revised the Introduction and Conclusion to clarify that our data-efficiency discussion refers to the overall training-data requirements of the TTS system. We have also removed the paragraph in Section 5.4 that could be interpreted as claiming that 65 hours of expressive speech alone demonstrates data efficiency. The role of the 65-hour expressive dataset is now described only in the context of tokenizer training.

Question 1: Evidence of Disentanglement. Does the reduction in WER/CER necessarily indicate that speaker and emotion information has been effectively removed from the discrete speech tokens? Could these improvements instead result from the information bottleneck reducing token complexity or producing a distribution that is easier for the language model to predict?

Response: We thank the reviewer for raising this important point. We agree that improvements in WER/CER alone cannot establish that speaker and emotion information have been effectively disentangled from the discrete tokens.

As described in our common response, we have added probing experiments in Section 5.1 to directly measure the speaker and emotion information retained in the tokens. By comparing our tokenizer with a No-prompt variant that shares the same architecture and training configuration but does not use decoder prompt conditioning, we show that prompt conditioning further reduces the recoverable speaker and emotion information. This provides direct evidence that the improvements are not solely due to a more predictable token representation.

Furthermore, in Table 8 of the revised paper, we have added a comparison with the Vevo tokenizer, which also employs an information bottleneck but does not incorporate our proposed prompt-conditioned disentanglement mechanism. Under the same downstream training and inference setting, our tokenizer achieves better cross-lingual speaker similarity and overall speech quality. This comparison provides additional evidence that the benefits of our approach go beyond those of using an information bottleneck alone.

Question 2: Relationship Between Disentanglement and Speaker Similarity. In Table 2, the average WER/CER decreases substantially from 8.00 to 6.16, while speaker similarity (SIM) increases only marginally from 0.529 to 0.539. If disentanglement reduces speaker and emotion information that may overlap with the separate conditioning embeddings in the flow-matching model, might a more noticeable improvement in SIM be expected? Could the authors explain why linguistic accuracy improves substantially while SIM remains nearly unchanged?

Response: We thank the reviewer for this insightful question. We believe that the different trends in WER/CER and SIM reflect the different roles of disentanglement in the language model (LM) and flow-matching model.

The improvement in WER/CER can be directly attributed to the benefits of disentanglement for the LM. By reducing speaker-related information in the discrete tokens, the tokenizer allows the LM to focus more on modeling linguistic content. Since the LM is primarily trained to predict speech tokens and is better suited to modeling semantic and linguistic information than global attributes such as speaker identity, this can lead to substantial improvements in linguistic accuracy.

In contrast, speaker similarity depends not only on disentanglement but also on how speaker information is incorporated into the flow-matching model. In our current system, the flow-matching model is conditioned only on a speaker embedding and prompt spectrum to preserve speaker identity. We acknowledge that this conditioning mechanism may be relatively limited in explicitly modeling and reconstructing speaker characteristics. Consequently, although disentanglement reduces speaker information in the generated tokens, the benefit may not translate into a similarly substantial improvement in SIM.

This suggests that a stronger speaker-conditioning mechanism in the flow-matching model could further improve speaker identity preservation. We consider this an important direction for future work.

Question 3: Data Efficiency of the Proposed Tokenizer The paper emphasizes using approximately 65 hours of expressive speech during disentangled tokenizer training. Could the authors clarify whether the proposed approach offers a data-efficiency advantage over conventional speech tokenizers or alternative disentanglement approaches under comparable expressive-data budgets? Such evidence would help assess whether the reported data efficiency is a distinctive benefit of the proposed method.

Response: Thank you for this question. This point is closely related to the concern raised in the Weakness 2. Please refer to our response to that concern for clarification.

Reviewer_dkdwOpenReview ↗

Official Comment by Reviewer dkdw

Thank you for the comprehensive responses and the substantial revisions to the manuscript. I sincerely appreciate the authors' efforts to address the reviewers' concerns through additional experiments, analyses, and clarifications.

In particular, I appreciate the newly added linear probing experiments. The reduction in classification accuracy provides useful evidence supporting the proposed disentanglement approach. However, I would appreciate some further clarification regarding the interpretation of these results.

  1. Interpretation of Emotion Probing Results

The authors suggest that the smaller reduction in emotion probing accuracy may be attributed to emotional cues varying across speech segments, unlike the relatively stable nature of speaker identity. While I understand that fine-grained emotional expressions may vary over time, I would expect utterance-level emotion information to remain distinguishable to some extent. Furthermore, the proposed flow-matching model explicitly uses utterance-level emotion embeddings for conditioning. Could the authors clarify why the temporal variation of emotional cues would explain the relatively small reduction in utterance-level emotion probing accuracy, and how this interpretation is supported by the current experiments?

  1. Interpretation of Classification Accuracy

Since the linear probes predict speaker identities and emotions from a fixed set of labels, could the authors clarify whether the reduction in accuracy reflects decreased discriminability across classes or a tendency toward particular predicted classes? Providing confusion matrices or class-wise results would help clarify how the probing performance changes and support the interpretation of the observed reduction in classification accuracy.

Reviewer_EhwiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

X-EMO is a cross-lingual zero-shot TTS system built around a disentangled speech tokenizer. They also build an AR LLM to predict tokens using a fixed native-speaker anchor utterance for cross-lingual prompting, and a flow-matching decoder injects CAM++ speaker and Emotion2Vec emotion embeddings from the prompt.

Critical strengths

The cross-lingual stability result is a great point and practical. WER into Indonesian stays around 3.1 regardless of prompt language, while much larger systems degrade badly, achieved efficiency than baselines. These results make the data-efficiency story credible, which is the paper's stated motivation.

Critical weaknesses

The central disentanglement claim is not directly tested. There is no probing experiment measuring speaker or emotion information in the tokens. Since the conditional-bottleneck idea is a fairly direct extension of previous works, style conditioning and the rest of the pipeline are standard. This missing evidence falls exactly on the part that carries the novelty.

X-Emo conditions on emotion2vec embeddings and emotion preservation is scored by an emotion2vec classifier, so the system is graded on reproducing the feature it consumes while baselines are not. No human listening test is reported anywhere, and speaker similarity is consistently the weakness of most systems.

Questions for authors

Report emotion preservation with a classifier independent of Emotion2Vec, or a small listening test. How much of the Table 4 gap survives?

How are speech and conditions pairs constructed for the non-emotional corpora, is same emotion simply assumed from same speaker?

3 comments

Weakness 1: The central disentanglement claim is not directly tested. There is no probing experiment measuring speaker or emotion information in the tokens.

Respond: We thank the reviewer for this comment. We have addressed this concern by adding a probing experiment in Section 5.1, which directly measures the speaker and emotion information retained in the discrete tokens. Please refer to our general response for the experimental details and results.

Weakness 2 and Question 1: X-Emo conditions on emotion2vec embeddings and emotion preservation is scored by an emotion2vec classifier. No human listening test is reported anywhere. Report emotion preservation with a classifier independent of Emotion2Vec, or a small listening test. How much of the Table 4 gap survives?

Respond: We thank the reviewer for this suggestion. As described in our general response, we have added an independent emotion classifier evaluation in Table 9 and released a demo page with audio samples to facilitate qualitative assessment.

We also considered conducting a listening test. However, given our four generation languages (Chinese, English, Malay, and Indonesian), it is challenging to recruit listeners who are proficient in both the source and target languages, particularly for Chinese–Malay and Chinese–Indonesian language pairs. Such bilingual proficiency is important for reliably assessing emotion preservation in cross-lingual speech generation. We therefore provide the independent classifier results and audio samples as complementary evidence for the quality of our generated speech.

Question 2: How are speech and conditions pairs constructed for the non-emotional corpora, is same emotion simply assumed from same speaker?

Respond: For non-emotional corpora, we assume all utterances are neutral and therefore do not explicitly model emotion consistency. We only enforce speaker consistency, which is ensured using the speaker labels provided in the corpus.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts, particularly the new experiments and independent emotion evaluation, which address several of my original concerns.

However, I still have two concerns. Although the proposed tokenizer reduces speaker information in the discrete tokens, the downstream speaker similarity improvements over the No-prompt baseline are marginal (0.01). This raises the question of whether the improved disentanglement translates into meaningful gains (WER and SpkSim matters). in zero-shot TTS. Furthermore, while speaker probing accuracy decreases substantially, emotion probing shows marginal differences, providing limited evidence for effective emotion disentanglement.

Overall, I will maintain my original rating.

We sincerely thank the reviewer for the follow-up and for acknowledging our additional experiments and independent emotion evaluation. We would like to briefly respond to the two remaining concerns.

1. The downstream speaker similarity improvements over the No-prompt baseline are marginal We agree that the improvement in speaker similarity over the No-prompt tokenizer is small. However, the improved disentanglement does translate into meaningful gains in intelligibility: WER/CER decreases in all four directions, with relative reductions of 7.3% (en→en), 18.7% (zh→zh), 11.3% (en→zh), and 11.2% (zh→en), while speaker similarity is also slightly improved in all directions.

The limited change in speaker similarity is expected from our architecture. As we responded to Reviewer dkdw, the WER/CER improvement mainly comes from the benefit of disentanglement for the language model. With less speaker information in the discrete tokens, the language model can focus on linguistic content, which it is better suited to model than global attributes such as speaker identity. On the other hand, speaker similarity depends not only on disentanglement but also on how speaker information is injected into the flow-matching model. In our current system, this relies only on a speaker embedding and the prompt spectrum, which may be limited in reconstructing fine-grained speaker characteristics. As a result, the reduced speaker information in the tokens does not translate into a comparably large SIM improvement.

2. Emotion probing shows marginal differences, providing limited evidence for effective emotion disentanglement. We agree that the reduction in emotion probing accuracy is much smaller than that in speaker probing. As discussed in our response to Reviewer dkdw, we see two possible explanations for this difference. First, emotion is only partially captured by utterance-level attributes; part of it is realized through local, content-dependent prosodic cues that the conditioning utterance cannot supply, so these cues remain in the tokens. Second, only approximately 65 hours of emotional speech are included in tokenizer training, which is small compared with the large and diverse set of speakers. The tokenizer therefore has far fewer opportunities to learn to disentangle emotion to the conditioning utterance than speaker identity.

We would also clarify that the main focus of this work is to learn disentangled speech tokens for cross-lingual zero-shot expressive TTS. In our framework, emotion preservation is mainly achieved through the explicit emotion conditioning in the acoustic model, whose effectiveness is supported by the emotion preservation results in Section 5.4 and the independent emotion evaluation. A more effective disentanglement of emotion at the token level, together with scaling up expressive training data, is an interesting direction for future work. In our revised manuscript, we state that the tokenizer substantially reduces speaker information and only partially reduces emotion information.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Direct Evidence for Disentanglement

We thank all reviewers for their constructive feedback. Several reviewers pointed out that our disentanglement claim was supported mainly by downstream TTS metrics, which do not directly reveal what information is retained in the discrete tokens. We agree and have added Section 5.1 "Probing Speaker and Emotion Information in Discrete Tokens," to provide direct evidence of disentanglement.

We train linear probes to predict speaker identity and emotion from utterance-level representations of discrete tokens, obtained by concatenating the temporal mean and standard deviation of quantized code vectors. Experiments are conducted on two unseen corpora, EmoV-DB (English) and CASIA (Chinese) We compare three representations: continuous HuBERT features, our tokenizer, and a No-prompt variant with the same tokenizer architecture and training configuration but without decoder prompt conditioning. Lower probe accuracy indicates less recoverable attribute information. We also evaluate downstream TTS performance using the No-prompt tokenizer under the same 5k-hour training regime. The results are shown in Table 1 of the revised paper:

Table: Effect of the prompt conditioning in tokenizer training. "HuBERT" denotes the original continuous features. "No-prompt" denotes the tokenizer identical to ours except that no prompt is provided to the decoder. Bold indicates the best result.

Task (test set) / DirectionHuBERTNo-promptOurs
Linear probe accuracy (%) ↓
Speaker (EmoV-DB)94.974.765.2
Speaker (CASIA)82.459.751.6
Emotion (EmoV-DB)96.077.876.7
Emotion (CASIA)78.444.844.5
Downstream TTS: WER or CER / SIM
en→en–7.58 / 0.5637.03 / 0.567
zh→zh–6.05 / 0.7034.92 / 0.705
en→zh–7.07 / 0.4006.27 / 0.405
zh→en–7.21 / 0.4686.40 / 0.479
The results provide direct evidence that decoder prompt conditioning reduces both speaker and emotion information retained in the discrete tokens. The reduction is more pronounced for speaker information than for emotion information. This difference is consistent with our observations in the original manuscript, where speaker information was more effectively disentangled than emotion information. This difference may be attributed to the fact that speaker identity is a relatively stable utterance-level attribute, whereas emotional cues may vary across speech segments.
AuthorsOpenReview ↗

Independent Emotion Evaluation and Audio Samples

Reviewers raised two related concerns regarding our emotion evaluation. First, since X-EMO is conditioned on Emotion2Vec embeddings and evaluated using an Emotion2Vec classifier, the reported gains might be affected by this overlap. Second, the original submission did not provide audio samples for qualitative assessment. We agree with both concerns and have made the following additions in the revised manuscript.

To address the potential evaluation bias caused by using Emotion2Vec for both conditioning and evaluation, we have added results using an independent emotion classifier in Table 9 of the revised paper. Specifically, we use the SpeechBrain wav2vec 2.0 model fine-tuned on IEMOCAP, which predicts four emotion classes: neutral, angry, happy, and sad. As shown in Table 9, the results obtained with this independent classifier are consistent with those reported using Emotion2Vec in Table 5. Our model maintains a comparable advantage over the baselines, suggesting that the observed improvements are not specific to the use of Emotion2Vec for evaluation.

We have also released an anonymous demo page: https://iclr2027random-droid.github.io/x-emo-demo/, which is linked in the abstract of the revised manuscript. The demo provides side-by-side audio comparisons between our model and other systems under the same prompts and target texts, allowing reviewers to directly compare their emotion preservation and speaker similarity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.