acceptodds
Under review as a conference paper at ICLR 2027

Emotional Enhancement of Cloned Voices via Speaker-Embedding Editing

Abstract

Native emotion instructions guide speech planning, but do not directly specify how the acoustic condition derived from a reference voice should change. We introduce a lightweight, reference-conditioned speaker-embedding editor for enhancing emotional expression in frozen text-to-speech synthesis. Trained on paired synthetic embeddings, the editor predicts a residual from the reference embedding and requested emotion, allowing both direction and magnitude to adapt to the reference rather than applying a shared shift. A scalar controls editing strength, and the edited condition enters CosyVoice 3 through its native acoustic interface without updating the synthesizer or requiring target-emotion recordings at inference. By holding speech tokens, acoustic prompts and decoding noise fixed, the method enables acoustic realization to be adjusted independently of changes to the speech plan. Experiments across Chinese and English, in both zero-shot and native emotional-instruction settings, show approximately 6.3–18.2% relative gains in mean target-emotion cosine. Bilingual listening evaluation further supports improved emotion appropriateness and conditional naturalness non-inferiority, alongside reduced speaker similarity at higher editing strength. These findings demonstrate a complementary approach to emotional enhancement that combines reference-dependent editing with adjustable control through a pretrained acoustic interface.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech synthesis, speaker embedding, emotion editing, frozen TTS, controllable speech

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_L5dSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a reference-conditioned speaker-embedding editing method to enhance emotional expression in a pretrained text-to-speech (TTS) system without retraining the synthesizer. Specifically, the authors train a FiLM-based residual predictor on paired synthetic neutral and target-emotion embeddings. Given a normalized CAM++ embedding extracted from a neutral reference recording and a target emotion label, the predictor estimates a reference-dependent residual. The predicted residual is applied with a scalar editing strength to modify the speaker condition supplied to the Flow-based acoustic model of CosyVoice 3 (CV3), while keeping the speech-token plan fixed across editing conditions and the pretrained synthesis components frozen.

Critical strengths

S1. Leveraging a Pretrained TTS Model for Emotional Enhancement

This paper proposes a speaker-embedding editing method to enhance emotional expression while keeping the pretrained CV3 synthesizer frozen. By keeping the speech-token plan fixed across editing conditions and reusing the existing acoustic conditioning interface, the method requires training only a lightweight FiLM-based residual editor without updating the pretrained synthesis components. This approach is a meaningful strength, as it enables additional emotion control while leveraging the synthesis capabilities of a pretrained TTS model.

Critical weaknesses

W1.Insufficient Evidence for the Distinct Benefits of Reference-Conditioned Editing

Although FiLM achieves higher target-emotion cosine than Shared residual, further analysis is needed to establish its distinct benefits beyond numerical improvements. In particular, its advantage over Shared residual largely disappears when residual norms are matched in Table 2, leaving the contribution of reference-dependent direction prediction unclear. The paper also discusses related methods such as EmoKnob and EmoSteer-TTS, but does not provide direct experimental comparisons with these approaches [1]. Furthermore, the proposed method requires training on synthetic neutral–target embedding pairs, while the main inference evaluation is limited to neutral references. Given these constraints, further evidence is needed to clarify the practical advantages of reference-conditioned editing beyond the reported gains over native emotional instructions, and to better position it relative to existing emotion-control approaches [2,3], particularly in terms of emotional expressiveness and speaker similarity.

[1] Wang, Siyi, et al. "Cocoemo: Composable and controllable human-like emotional tts via activation steering." arXiv preprint arXiv:2602.03420 (2026). [2] Um, Se-Yun, et al. "Emotional speech synthesis with rich and granularized control." ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020. [3] Li, Tao, et al. "Cross-speaker emotion disentangling and transfer for end-to-end speech synthesis." IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (2022): 1448-1460.

W2. Trade-off Between Emotional Enhancement and Speaker Similarity

The proposed method improves emotional expression at higher editing strengths, but this comes at the cost of reduced speaker similarity. In Table 3 and Appendix C, speaker-similarity MOS decreases by 0.448 in Chinese and 0.224 in English at α = 5. While the authors acknowledge this trade-off and do not claim to disentangle speaker identity from emotion, preserving speaker characteristics remains essential for voice cloning. Therefore, it is important to establish whether the proposed method achieves better emotional expression than existing editing approaches at comparable speaker-similarity levels. Without such a comparison, its practical advantage remains insufficiently demonstrated.

W3. Lack of Accessible Audio Demonstrations

Although the paper reports perceptual evaluation results in Table 3, no accessible audio demo page or sample links are provided in the manuscript.

Questions for authors

Q1. Perceptual Comparison with Shared Residual Editing

Table 3 reports listening results only for unedited CV3 and FiLM, whereas Table 2 also includes Shared residual and norm-matched FiLM. Could the authors clarify why these baselines were excluded from the listening evaluation?

Q2. Details of Synthetic Training Data Construction

Could the authors provide more details on the construction of the 195,301 synthetic neutral–target embedding pairs, including the number of samples per target emotion, the speech-generation model and synthesis settings, and the source and composition of the input texts?

Q3. Applicability to Non-Neutral References

The authors acknowledge that the proposed method has not been validated for non-neutral references. Could the authors further clarify the expected behavior of the current editor when applied to an already emotional reference, such as Angry → Happy? In particular, is there any preliminary evidence indicating whether the learned residuals remain meaningful outside the neutral-reference setting?

2 comments
Reviewer_L5dSOpenReview ↗

Official Comment by Reviewer L5dS

Thank you for the detailed responses and clarifications.

Regarding Q3, I understand that the proposed method has been evaluated only on neutral references and that its applicability to non-neutral references remains unvalidated. (Section 6 — Limitations: Adaptive dosing, arbitrary emotional references, arousal control and cross-backbone transfer of the trained editor remain unvalidated.) My question was not intended to suggest that the paper claims such generalization, but rather to better understand the practical scope of the approach and whether the authors had any preliminary evidence or insights to share.

I also appreciate the additional experiments reported in your responses to other reviewers, particularly the improvements achieved by V2 over dose-tuned shared editing. However, given that the original FiLM rule performs worse than dose-tuned shared editing at matched loss, I am not fully convinced that the V2 results alone sufficiently address my original concerns. It would be helpful to provide further analysis and clarification of this performance difference, as well as how the additional evidence from V2 relates to the central contributions of the original paper.

Thank you again for your thoughtful responses and additional efforts.

Reviewer_L5dSOpenReview ↗

Official Comment by Reviewer L5dS

Thank you for the additional experiments and detailed clarifications. I appreciate the authors' continued efforts to address my concerns.

The new reference-averaged control provides useful evidence that the learned reference-dependent predictions contribute to the improvements achieved by V2. However, while combining the learned and shared directions improves performance, I remain cautious about interpreting this result as evidence of strong methodological novelty or the superiority of the original FiLM approach. I will consider the additional evidence alongside the perspectives of the other reviewers before finalizing my assessment.

Thank you again for your thoughtful responses and additional efforts.

Reviewer_hZw9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper strengthens emotion in cloned voices by editing the speaker embedding of a frozen CosyVoice 3. A FiLM network maps a neutral reference's embedding and a target emotion to a residual, added at an adjustable strength. Because this embedding conditions the acoustic model rather than the token planner, the speech-token plan and prompt stay fixed. Trained on synthetic pairs, the editor raises an emotion2vec proxy on 18 ESD speakers at a measurable cost in speaker similarity. In a bilingual listening study with 16 respondents, the edits raise perceived emotion and naturalness meets a conditional non-inferiority criterion. Similarity falls at the higher dose and, in Chinese, fails that criterion even at the lower one. The stated difference from EmoKnob is reference-conditioned residual learning.

Critical strengths
  1. Controllable emotional enhancement through the model's native interface: Editing only the speaker condition keeps the token plan, prompt and decoding noise fixed, so the effect of the edit can be isolated. Trained only on synthetic pairs, the editor is applied to real neutral references without fine-tuning and adds expression even on top of CosyVoice 3's own emotion instruction. This capability holds even without the reference-conditioning claim, and I see it as the paper's most useful contribution.

  2. A listening study that reports costs as well as gains: Emotion, intensity, naturalness and similarity are rated separately. The expression gains stay positive under respondent resampling, and the similarity losses are reported (Tables 7–8). Within what the authors label an exploratory analysis, this supports an audible gain at a measured cost, which a control method needs to show.

Critical weaknesses
  1. For generated speech, the paper does not establish a benefit of reference-conditioned learning over a dose-tuned shared edit. This is the paper's first listed contribution and its stated difference from EmoKnob. The held-out embedding result (cosine 0.909 vs. 0.888 for a pooled residual) supports it only at the prediction level. In generated speech:
  • Scaled to the shared residual's length, FiLM trails the shared edit on emotion in three of four groups. Unscaled, it gains more emotion but loses more similarity in every group, consistent with a larger edit (Table 2).
  • In the two-speaker dose grid, at comparable speaker similarity, FiLM and the shared "Mean direction" edit are close in three of four blocks, and FiLM leads in the fourth (Tables 16–17). The shared edit was tested only up to α = 1.5, never reaching FiLM's emotion scores at α = 1, and each table covers one speaker.
  • The listening study has no shared-edit arm.
  1. The authors acknowledge this gap (Figure 1) and describe the supported contribution as "learned, adjustable enhancement through the native realization interface". That is a modest step beyond EmoKnob-style editing, which already sets strength by hand; a shared edit also keeps the plan fixed. A clear advantage over a dose-tuned shared residual on the 18-speaker panel at matched similarity cost or emotion gain, ideally also in listening tests, would raise my rating to weak accept. Either direction or magnitude may supply that advantage.
Questions for authors
  1. Can the authors confirm from rendering records which editor checkpoint produced the listening stimuli, and that tokens, prompts and noise were matched across doses? Appendix C reconstructs this mapping rather than verifying it, and the perceptual evidence rests on it.
  2. Appendix L.6 says the method's organization is shared with the unpublished PATHOS-TTS manuscript and that sample overlap has not been audited. Which methods, data and listening materials are shared, and what is unique to this submission? An anonymized copy would help.
  3. How was the main 195,301-pair training set produced (generator, source speech and text, prompts, filtering)? Evaluation speakers' presence in editor training is not established, so this matters for reproducibility and for the synthetic-to-real claim.
Reviewer_BNbhOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a speaker-embedding editing method to improve emotional expression in voice cloning. Using CosyVoice 3 as a frozen synthesizer, the authors train an editor to predict an emotion-dependent residual from a reference speaker embedding and a target emotion label. Objective and subjective evaluations suggest improved emotional expression, although stronger editing also reduces speaker similarity.

Critical strengths

The paper addresses a relevant problem: improving emotion control in voice cloning while keeping the pretrained synthesizer frozen. The evaluation includes both objective metrics and listening tests in Chinese and English.

Critical weaknesses

The innovation appears incremental compared with existing speaker-embedding editing methods such as EmoKnob. Although the paper includes a shared-residual baseline, it does not directly compare against EmoKnob. Moreover, the norm-matched results do not clearly demonstrate an advantage from the proposed reference-conditioned editing direction over the simpler shared residual. A direct comparison under matched settings would help establish the contribution.

The paper also lacks important details about how the paired synthetic training data were prepared. It is unclear which synthesis model was used, how the neutral and emotional samples were generated and paired, and how the emotion labels were validated. These details are important for reproducibility and for assessing the quality of the training targets.

Questions for authors
  1. How were the paired neutral and emotional training samples generated? Which synthesis model, reference recordings, and emotion controls were used, and were speaker identity and text matched within each pair?

  2. The final prediction averages the outputs of three editors trained with different random seeds. Why is this ensemble used, and how does its performance compare with that of each individual editor?

  3. Could the authors compare the proposed method directly with EmoKnob under the same synthesis and evaluation settings?

Reviewer_ZKt7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper learns a reference-conditioned residual editor for the speaker embeddings used by frozen CosyVoice 3. The editor is trained on synthetic neutral–emotional embedding pairs and changes the acoustic condition while keeping speech tokens, prompts, and decoding noise fixed. Experiments on Chinese and English ESD references compare learned editing with unedited synthesis, shared residuals, and norm-matched controls. Automatic scores and listening ratings indicate stronger emotional expression, although higher editing strength reduces speaker similarity, and the advantage over appropriately scaled shared editing remains unclear.

Critical strengths

S1: Holding speech tokens, acoustic prompts, and decoding noise fixed makes the objective comparison informative about the effect of changing the speaker condition itself.

S2: The reported emotion-appropriateness gains of 0.354 in Chinese and 0.690 in English at α = 5 provide perceptual evidence beyond embedding scores, with speaker-similarity losses reported alongside them (Table 7).

Critical weaknesses

W1: Norm-matched FiLM falls below the shared residual in three of four settings and exceeds it by only 0.0003 in the fourth, leaving its advantage over a strength-tuned shared edit unproven (Table 2).

W2: Appendix C verifies reconstructed stimulus identifiers but not the original audio-to-checkpoint mapping, leaving the connection between the listening results and the stated generation settings insufficiently documented.

Questions for authors

Q1: At matched speaker-similarity loss, does FiLM outperform a shared residual with tuned strength or a learned reference-dependent scalar?

Q2: Can the original listening waveforms be traced to their exact editor checkpoints, cached tokens, acoustic prompts, and decoding seeds?

Q3: Do the emotion-gain and naturalness non-inferiority conclusions hold when speaker and listener variation are modeled jointly?

Q4: Which synthesis models, prompts, source voices, and filtering rules were used to construct the 195,301 training pairs?

Comments on the paper

1 comment
Reviewer_BNbhOpenReview ↗

Training Data Construction

Thanks for your reponse.

Could you please clarify more detail on how you synthesize the training data? What model did you use? What is the input text, audio, and instructions to generate the training data pairs?

You mentioned B02 and B04, could you clarify these two data sets? Do you have these two data sets in the paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.