acceptodds
Under review as a conference paper at ICLR 2027

CoCiTTS: Towards Character-Level Speaker Consistency in Zero-Shot TTS for Cross-Lingual Cinematic Dubbing

Abstract

Cross-Lingual Cinematic Dubbing aims to synthesize target-language speech from source-language speech in movies or TV shows, which requires a character to retain a consistent voice across scenes with different emotional expressions. However, existing zero-shot text-to-speech (TTS) systems mainly optimize utterance-level speaker similarity and overlook character-level speaker consistency across multiple utterances. This limitation becomes particularly challenging in cinematic scenarios, where the same character frequently exhibits diverse emotional states, and emotion-rich reference utterances may lead to inconsistent speaker timbre across generated utterances. In this paper, we identify emotion leakage within speaker representations as an important factor contributing to speaker timbre drift. To address this issue, we propose CoCiTTS (Consistency-Oriented Cinematic Text-to-Speech), a zero-shot TTS framework that learns consistency-oriented speaker conditioning for cross-lingual cinematic dubbing. CoCiTTS introduces a Speaker Timbre Purification (STP) module based on adversarial disentanglement to suppress emotion leakage while preserving speaker identity. Furthermore, a Fine-grained Emotion Compensation (FEC) module models global and local emotional characteristics through an independent pathway, enabling expressive generation without contaminating speaker representations. In addition, we construct CineSpeech, an emotion-rich cinematic speech benchmark containing 136 characters, and establish consistency-aware evaluation methods for character-level timbre preservation. Extensive experiments on multilingual cross-lingual tasks demonstrate that CoCiTTS effectively improves speaker consistency while maintaining competitive speech quality, speaker similarity, and emotional expressiveness compared with existing zero-shot TTS systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-to-Speech, Cross-Lingual Cinematic Dubbing, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_U4RmOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoCiTTS, a zero-shot TTS framework designed to improve speaker consistency across emotionally diverse utterances in cross-lingual cinematic dubbing. Built on IndexTTS2, the method introduces Speaker Timbre Purification (STP) to disentangle emotion from speaker representations and Fine-grained Emotion Compensation (FEC) to preserve emotional expressiveness. The authors also introduce CineSpeech, a benchmark of 950 utterances from 136 characters, along with speaker consistency metrics. Experiments across multiple cross-lingual synthesis tasks demonstrate improved speaker consistency over existing zero-shot TTS baselines, supported by ablation and emotion-control analyses.

Critical strengths

Consistent empirical improvements across multiple settings

The proposed method improves speaker consistency across four cross-lingual synthesis tasks on both CineSpeech and CineDub-CN. The additional experiments applying STP to CosyVoice also provide some evidence that the approach can be transferred to another TTS backbone. These results suggest that the proposed training strategy can be practically useful for improving speaker consistency in zero-shot TTS systems.

Useful analysis of emotion-related speaker variations

The paper investigates the relationship between emotional variations in reference speech and speaker consistency. In particular, the emotion-controlled intervention experiment in Appendix A.2.2 provides supporting evidence that changing the emotional characteristics of speaker references can affect the consistency of generated speech. Although the underlying phenomenon is not fundamentally new, the analysis is relevant to expressive zero-shot TTS.

Critical weaknesses

Limited methodological novelty

The proposed framework primarily combines established techniques within IndexTTS2. STP uses gradient reversal-based adversarial disentanglement and ArcFace speaker classification, while FEC adds Conformer-based local emotion representations and cross-attention to the existing global emotion conditioner. Despite emphasizing character-level speaker consistency, the method does not explicitly model cross-utterance relationships or introduce a character-level consistency objective. Instead, it relies on utterance-level speaker–emotion disentanglement. Although the problem formulation and CineSpeech benchmark are useful, the methodological contribution remains largely incremental, with limited distinction from existing disentanglement and emotion-conditioning approaches.

Insufficient isolation of the proposed components' contributions

The experiments demonstrate improvements in speaker consistency through speaker similarity, subjective evaluations, and emotion-controlled interventions. However, the contribution of the proposed architecture is not fully isolated from additional backbone training. In Section 3.3, the first training stage jointly optimizes STP and the text-to-semantic LLM, whereas the main comparisons use backbone models without matched continued fine-tuning. Thus, it is unclear whether the reported improvements, particularly in WER and UTMOS, result from the proposed disentanglement mechanism or additional backbone optimization. Furthermore, although Figure 4 demonstrates the benefit of FEC, it does not isolate the contribution of the new local emotion branch from the existing global emotion conditioner. A controlled comparison retaining global conditioning while removing the local branch would better establish its necessity. Overall, the reported results support the effectiveness of the combined system, but provide limited evidence that its individual architectural components offer advantages beyond established techniques and continued backbone training.

Questions for authors
  1. How does STP differ from existing adversarial speaker–emotion disentanglement methods? What is the main technical contribution beyond integrating established techniques into IndexTTS2?

  2. Can the authors compare against IndexTTS2 with matched training data, steps, and trainable backbone parameters, but without STP and FEC, to isolate the effects of the proposed modules?

  3. Since SC measures only intra-character similarity, how do the authors distinguish improved consistency from genuine identity preservation? Can they provide a perceptual identity evaluation beyond the reported SS and CMOS metrics?

  4. Can the authors compare STP with global-only emotion conditioning against STP with the proposed local emotion branch to establish its additional benefit?

1 comment

Thank you for the additional controls and clarification. They address my concerns about continued-training effects and the local branch’s contribution. However, the limited methodological novelty remains a concern. I am raising my rating from 1 (Clear rejection) to 2 (Weak rejection).

Reviewer_wW5bOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that zero-shot TTS for dubbing should be evaluated on character-level timbre consistency across emotionally diverse references rather than per-utterance speaker similarity. Authors add two modules to IndexTTS2 with proposed consistency metrics, builds a evaluation set (CineSpeech) from 50 movies and TV shows, and reports gains on CineSpeech and CineDub-CN.

Critical strengths

The problem formulation is the core contribution. Character-level consistency under emotional variation is a real blind spot of current zero-shot TTS evaluation, and the diagnostic evidence is solid.

Critical weaknesses

It is unclear how much of the reported improvement comes from the proposed method itself. The IndexTTS2-ML baseline does not appear to undergo the additional ESD fine-tuning used by CoCiTTS. Since this training could independently affect SS, SC, the current comparison provides insufficient evidence that these gains are driven by STP and FEC.

The method itself is an assembly of standard parts. GRL-based adversarial removal of a nuisance attribute from speaker representations is a standard recipe that has appeared repeatedly in the cross-speaker emotion transfer literature the paper cites, and FEC is ordinary attention pooling. Moreover, the backbone IndexTTS2 already advertises independent speaker/emotion control, so this is a patch on an existing disentanglement mechanism. The novelty is limited.

The emotion evaluation also needs clarification. STP is trained on 5 emotion categories from ESD, whereas CineSpeech includes 6 emotions The paper does not explain how these emotion labels were obtained or how this mismatch affects the analysis in Table 2.

Questions for authors

The claim that FEC preserves emotional expressiveness is not fully supported by the objective results. On CineSpeech, ES remains at or below the backbone across all four language directions. The results support improved consistency with relatively limited degradation in emotion similarity

Reviewer_pumDOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies character-level voice consistency in cross-lingual cinematic dubbing when reference utterances exhibit different emotions. The authors argue that emotion leakage in speaker representations causes timbre drift and propose CoCiTTS on top of IndexTTS2. Its Speaker Timbre Purification (STP) module combines adversarial emotion classification through a gradient reversal layer with ArcFace speaker classification, while Fine-grained Emotion Compensation (FEC) models emotion through a separate global-local pathway. The paper also introduces CineSpeech, containing 950 utterances from 136 characters, together with SC/SCR consistency metrics. Across four Chinese-to-other-language tasks, CoCiTTS generally improves objective and subjective consistency over its backbone while largely preserving intelligibility, quality, and emotion similarity.

Critical strengths
  1. The paper clearly distinguishes utterance-level speaker similarity from character-level consistency across scenes and emotions. The emotion-probing and controlled-intervention experiments provide useful evidence that this is a practically important issue, especially for cinematic dubbing where evaluating isolated utterances alone is insufficient.

  2. The evaluation covers four target languages, two cinematic datasets, both objective and subjective metrics, and an additional CosyVoice backbone. Together with the STP/FEC ablation, these results suggest that the observed consistency gains are not confined to one language or architecture. CineSpeech may also become a useful evaluation resource.

Critical weaknesses
  1. SC only measures pairwise similarity among generated utterances of the same character, so producing more neutral or homogenized voices can also increase it. Average SS and a t-SNE plot for 15 speakers do not establish that identities remain separated across characters. Quantitative inter-character measures, such as identification accuracy, EER/minDCF, or an inter/intra-class margin, are missing, as is quantitative robustness across multiple speaker encoders. Moreover, SCR divides the SC of target-language synthetic speech by that of Chinese movie recordings, despite differences in language, noise, and channel. Its interpretation as a “retention” rate is therefore questionable; the 106.78% result for CineSpeech zh-de may indicate over-smoothing rather than improved identity preservation.

  2. Stage 1 updates the text-to-semantic LLM in addition to STP, but there is no matched continued-training baseline using the same data, steps, and trainable modules while removing only GRL/ArcFace. Consequently, the WER, UTMOS, and consistency gains in Table 3 cannot be cleanly attributed to the proposed modules. The claimed fine-grained local emotion modeling also lacks FEC-only, global-only, local-only, and standard-pooling comparisons and is evaluated only by utterance-level ES. Finally, the experiments report no repeated runs, confidence intervals, or significance tests, while the CMOS pairing protocol and listener language proficiency are unspecified.

Questions for authors
  1. Do the IndexTTS2/IndexTTS2-ML baselines use exactly the same training data, update steps, and trainable modules as CoCiTTS? Please provide a matched continued-training baseline and variance over repeated runs.
  2. What are the results for global-only, local-only, and combined FEC variants? Can frame- or segment-level emotion evaluation demonstrate that the local branch preserves temporal variation rather than merely improving utterance-level emotion similarity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.