- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CoCiTTS, a zero-shot TTS framework designed to improve speaker consistency across emotionally diverse utterances in cross-lingual cinematic dubbing. Built on IndexTTS2, the method introduces Speaker Timbre Purification (STP) to disentangle emotion from speaker representations and Fine-grained Emotion Compensation (FEC) to preserve emotional expressiveness. The authors also introduce CineSpeech, a benchmark of 950 utterances from 136 characters, along with speaker consistency metrics. Experiments across multiple cross-lingual synthesis tasks demonstrate improved speaker consistency over existing zero-shot TTS baselines, supported by ablation and emotion-control analyses.
Critical strengths
Consistent empirical improvements across multiple settings
The proposed method improves speaker consistency across four cross-lingual synthesis tasks on both CineSpeech and CineDub-CN. The additional experiments applying STP to CosyVoice also provide some evidence that the approach can be transferred to another TTS backbone. These results suggest that the proposed training strategy can be practically useful for improving speaker consistency in zero-shot TTS systems.
Useful analysis of emotion-related speaker variations
The paper investigates the relationship between emotional variations in reference speech and speaker consistency. In particular, the emotion-controlled intervention experiment in Appendix A.2.2 provides supporting evidence that changing the emotional characteristics of speaker references can affect the consistency of generated speech. Although the underlying phenomenon is not fundamentally new, the analysis is relevant to expressive zero-shot TTS.
Critical weaknesses
Limited methodological novelty
The proposed framework primarily combines established techniques within IndexTTS2. STP uses gradient reversal-based adversarial disentanglement and ArcFace speaker classification, while FEC adds Conformer-based local emotion representations and cross-attention to the existing global emotion conditioner. Despite emphasizing character-level speaker consistency, the method does not explicitly model cross-utterance relationships or introduce a character-level consistency objective. Instead, it relies on utterance-level speaker–emotion disentanglement. Although the problem formulation and CineSpeech benchmark are useful, the methodological contribution remains largely incremental, with limited distinction from existing disentanglement and emotion-conditioning approaches.
Insufficient isolation of the proposed components' contributions
The experiments demonstrate improvements in speaker consistency through speaker similarity, subjective evaluations, and emotion-controlled interventions. However, the contribution of the proposed architecture is not fully isolated from additional backbone training. In Section 3.3, the first training stage jointly optimizes STP and the text-to-semantic LLM, whereas the main comparisons use backbone models without matched continued fine-tuning. Thus, it is unclear whether the reported improvements, particularly in WER and UTMOS, result from the proposed disentanglement mechanism or additional backbone optimization. Furthermore, although Figure 4 demonstrates the benefit of FEC, it does not isolate the contribution of the new local emotion branch from the existing global emotion conditioner. A controlled comparison retaining global conditioning while removing the local branch would better establish its necessity. Overall, the reported results support the effectiveness of the combined system, but provide limited evidence that its individual architectural components offer advantages beyond established techniques and continued backbone training.
Questions for authors
-
How does STP differ from existing adversarial speaker–emotion disentanglement methods? What is the main technical contribution beyond integrating established techniques into IndexTTS2?
-
Can the authors compare against IndexTTS2 with matched training data, steps, and trainable backbone parameters, but without STP and FEC, to isolate the effects of the proposed modules?
-
Since SC measures only intra-character similarity, how do the authors distinguish improved consistency from genuine identity preservation? Can they provide a perceptual identity evaluation beyond the reported SS and CMOS metrics?
-
Can the authors compare STP with global-only emotion conditioning against STP with the proposed local emotion branch to establish its additional benefit?
Thank you for the additional controls and clarification. They address my concerns about continued-training effects and the local branch’s contribution. However, the limited methodological novelty remains a concern. I am raising my rating from 1 (Clear rejection) to 2 (Weak rejection).