acceptodds
Under review as a conference paper at ICLR 2027

ControlAnything: Unified word-level, Intensity-Aware, and Composable Control for Expressive Text-to-Speech

Abstract

Emotional expression in human speech varies across words, exhibits varying intensity levels, and often blends multiple affective states. Reproducing this flexibility in text-to-speech (TTS) requires joint control over where emotions occur, how strongly they are expressed, and how they are combined, while applications such as voice assistants further demand fast generation for responsive interaction. However, existing methods typically address these control dimensions separately, and fine-grained word-level control has been explored primarily in autoregressive (AR) TTS, leaving unified control in efficient non-autoregressive (NAR) architectures largely underexplored. We present ControlAnything, a unified framework for word-level, intensity-aware, and composable emotional control that preserves NAR inference efficiency. First, Soft-boundary Anchor Intervention (SAI) maps target text spans to acoustic regions and progressively reinforces temporal anchoring as acoustic representations evolve during generation. Second, Emotion Concept Modulation (ECM) learns distinct emotion-specific transformations within a shared modulation mechanism, enabling fine-grained intensity adjustment and flexible multi-emotion composition. Together, SAI and ECM support localized intensity adjustment and emotion composition across different pretrained TTS models. Extensive experiments demonstrate state-of-the-art controllability in word-level localization, emotion-intensity adjustment, and multi-emotion composition, while preserving speech quality and generation efficiency comparable to the baselines. Audio demonstrations are available at https://controlanything-review-demo.vercel.app/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text to speech, Audio processing, Speech language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6jChOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an emotion control framework for non-autoregressive speech synthesis. It supports three types of control: specifying which words carry emotions, how to adjust the intensity of emotions, and mixing multiple emotions within the same segment. Specifically, SAI maps the target words to corresponding time frames, limiting the location of emotional intervention. ECM controls the intensity and mixing ratio of emotions by scaling and combining the modulation parameters of different emotions. The results show that the proposed method can achieve local emotion control while making the transitions between segments more natural.

Critical strengths
  1. The three types of emotion control can be used in combination. ControlAnything not only can specify which words carry emotions, but also can simultaneously adjust the overall emotional intensity and the relative weights of various emotions on these words.
  2. The experiment is thorough. The authors not only evaluated the three control capabilities separately, but also tested the performance when all three were enabled simultaneously, and verified the roles of SAI and ECM through module ablation.
Critical weaknesses
  1. The method innovation of ECM is rather limited. ECM learns a set of scaling and offset parameters for each emotion, then controls the intensity by modulating the amplitude through scaling, and achieves emotion blending through weighted superposition. The existing ablation studies mainly demonstrate the usefulness of ECM within this framework, but they are not sufficient to show its advantages over more direct feature modulation schemes.
  2. The computational cost of word-level inference is not yet clear. The complete process includes one synthesis without emotion intervention, word-level alignment based on ASR, and two rounds of controlled synthesis. The paper does not clearly state whether the reported RTF takes into account these steps, making it difficult to fairly compare the inference efficiency with other methods.
Questions for authors
  1. Could you please clarify whether the RTF in Table 3 is calculated based on the entire word-level control process, or only accounts for the time consumption of the synthesized part?
  2. When conducting word-level comparison, how do you concatenate the generated baseline speech segments? Is it done by direct concatenation, or is some smoothing processing such as cross-damping applied?
  3. If ASR overestimates the start and end times of words, to what extent can the second alignment correct this deviation? Have you encountered situations where the emotion of the target word is not properly adjusted, or where adjacent words are also affected?
5 comments
AuthorsOpenReview ↗

Replies to Reviewer 6jCh: first part

We thank the reviewer for recognizing our unified framework for fine-grained emotional control and the comprehensive evaluation supporting its effectiveness. We address the reviewer's concerns below.

Question 1 and Weakness 2: Full-Pipeline Runtime for Word-Level Control

Thanks for these helpful comments. The reported word-level RTF of 0.137 includes the initial unsteered synthesis, one ASR-based alignment, and the first controlled synthesis. The second alignment and controlled synthesis in Eq. (8) constitute an optional refinement step that was not employed in our main experiments and is therefore excluded from the reported RTF.

We detail the RTF calculated formulations as below:

RTFsingle=T0+A1+T1D1.\mathrm{RTF}_{\mathrm{single}} = \frac{T_0 + A_1 + T_1}{D_1}.

This is the configuration used in Tables 1 and 3 of the main experiments, including the initial unsteered synthesis and one ASR-based temporal-anchor alignment.

RTFrefined=T0+A1+T1+A2+T2D2.\mathrm{RTF}_{\mathrm{refined}} = \frac{T_0 + A_1 + T_1 + A_2 + T_2}{D_2}.

This corresponds to the optional additional alignment refinement. Here:

  • T0T_0 is the time required for the initial unsteered synthesis.
  • T1T_1 and T2T_2 are the times required for the first and second controlled synthesis passes.
  • A1A_1 and A2A_2 are the times required for the first and second alignments, including ASR inference and temporal-anchor construction.
  • D1D_1 and D2D_2 are the final output audio durations for the single-pass and two-pass controlled-generation configurations, respectively.

The table below also reports speech generation RTF for conventional models as a speed reference.

Table A. Inference Efficiency and Control Performance.

MethodGeneration configurationControl Words SER (%) ↑RTF ↓
CosyVoice2Sentence-level generation—0.856
IndexTTS2Sentence-level generation—0.872
F5-TTSSentence-level generation—0.080
OursDefault sentence-level control, without SAI (Table 2)—0.087
OursDefault word-level control, with SAI (Table 1 and 3)0.6460.137
Ours + second alignmentComplete pipeline with a second alignment0.6510.174

These results clarify that the reported word-level RTF of 0.137 includes the initial unsteered synthesis, ASR alignment, and first controlled generation, rather than only the final synthesis pass. The optional refinement in Eq. (8) slightly improves Control Word SER from 0.646 to 0.651, while increasing RTF from 0.137 to 0.174. Both configurations remain faster than real time and substantially more efficient than CosyVoice2 and IndexTTS2 under the same hardware and evaluation conditions. We will clarify the inference pipeline and RTF calculation in the revised manuscript.

AuthorsOpenReview ↗

Replies to Reviewer 6jCh: second part

Question 2: Fairness of Comparisons with Segment-Wise Baselines

Thanks for this valuable comment. We clarify this issue from two perspectives.

1. Segment-wise Baseline Settings. Currently, no publicly accessible TTS models support native word-level emotion control. Therefore, we followed the segment-wise evaluation setting of WeSCon [1] for baselines such as F5-TTS-seg and IndexTTS2-seg. This setting approximates word-level control through segment-wise synthesis and concatenation.

We acknowledge that segment-wise synthesis may introduce boundary discontinuities, potentially disadvantaging these baselines in transition smoothness evaluations.

2. Native Word-level Comparisons. To address this concern, we contacted the authors of WeSCon and TED-TTS and just obtained partial code resources and technical assistance. Based on these materials and the implementation details provided in their papers, we reproduced both methods and conducted additional evaluations using the same training and test data as our method.

Table B. Word-level emotion control.

MethodWER (%) ↓DNSV-Pro ↓S-SIM ↑AutoPCP ↑Emo2v. ↑Aro. ↑
WeSCon4.3725.8140.8972.7140.5890.656
TED-TTS3.4766.7520.8912.6890.6040.689
Ours3.5625.0740.8832.8270.6460.733

As shown in Table B, our method achieves the lowest DNSV-Pro of 5.074 and the highest AutoPCP, Emo2v., and Aro. scores of 2.827, 0.646, and 0.733, respectively. These results demonstrate improved prosodic consistency and localized emotion control over native word-level methods, further supporting the effectiveness of our approach beyond comparisons with segment-wise baselines.

[1] Tianrui Wang et al. Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis. NeurIPS, 2025.

AuthorsOpenReview ↗

Replies to Reviewer 6jCh: third part

Question 3: SAI Robustness to Boundary Perturbations

Thanks for this helpful comment. We conducted an additional timestamp perturbation experiment to evaluate the robustness of SAI. We randomly perturbed the word boundaries across the entire test set, using perturbation ranges of ±20, ±50, and ±100 ms. We report the residual localization error rate, ER@50 ms: the interval in which control is actually applied is compared with independent manual annotations of the target-segment boundaries in the corresponding final output. A request is counted as an error if either the start or end boundary deviates by more than 50 ms. Missing target words and localization failures are also counted as errors. The results are shown below.

Table C. Robustness to ASR Boundary Perturbations.

Initial boundary perturbationSingle-pass ER@50 ms ↓With refinement ER@50 ms ↓Single-pass Emo2v. ↑With refinement Emo2v. ↑
No additional perturbation3.0%2.0%0.6460.650
±20 ms3.2%2.1%0.6430.649
±50 ms4.7%2.5%0.6310.646
±100 ms12.6%3.0%0.6010.638

With the original ASR timestamps, the default single-pass configuration already achieves a low localization error rate of 3.0%, which decreases further to 2.0% with optional refinement. As the perturbation magnitude increases, localization errors in the single-pass configuration gradually increase, while two-pass refinement reduces residual misalignment and better preserves the target emotional expression. These results indicate that SAI can tolerate small errors in the initial boundaries. When the initial localization error is larger or an application requires higher temporal precision, two-pass refinement can further reduce localization errors.

Weakness 1: ECM Design and Methodological Novelty

Thanks for this insightful comment. The main contribution of ECM lies in learning reusable emotion-specific modulation that support multiple control dimensions under a shared training objective. We clarify two key aspects below.

1. Unified Emotion Learning. Rather than introducing separate training objectives for intensity and composition, ECM learns distinct emotion-specific transformations using the backbone's original conditional flow-matching objective with emotion-category supervision. These learned transformations can subsequently be scaled for continuous intensity adjustment and combined for multi-emotion composition, without requiring intensity-specific annotations, mixed-emotion training targets, or dimension-specific retraining. Thus, the key contribution is not the scaling or weighted combination itself, but the ability to derive multiple control capabilities from a common set of learned emotion-specific transformations.

2. Effectiveness of Emotion-Specific Modulation. Our ablation study (Table 8) further supports this design. Replacing emotion-specific modulation with a single shared mapping for all emotions reduces H-Rate from 0.458 to 0.305 and N-MOS from 4.23 to 3.18, while increasing DNSV-Pro from 9.916 to 29.803. These results demonstrate the importance of learning distinct emotion-specific transformations rather than relying on a single undifferentiated mapping.

Overall, ECM provides a unified emotion-conditioning mechanism that supports categorical emotion expression, continuous intensity adjustment, and multi-emotion composition without separate training objectives for each control dimension.

Reviewer_KyCvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The comparison of intervention locations provides a useful practical insight. Intervening inside DiT blocks offers a balance between accurate placement and smooth transitions, compared with conditioning at the encoder or joining separately generated segments.
  • The experiments cover individual control dimensions and their combinations. The F5-TTS results show improved emotional expression and more natural speech than the segment-based baselines.
Critical strengths
  • The comparison of intervention locations provides a useful practical insight. Intervening inside DiT blocks offers a balance between accurate placement and smooth transitions, compared with conditioning at the encoder or joining separately generated segments.

  • The experiments cover individual control dimensions and their combinations. The F5-TTS results show improved emotional expression and more natural speech than the segment-based baselines.

Critical weaknesses
  • The methodological contribution appears incremental. Location, intensity, and composition are implemented through a temporal mask, residual scaling, and weighted combinations of emotion-specific transformations. These choices form a workable system, but the paper offers limited new understanding of how the three control dimensions should be learned or coordinated. The unified formulation mainly brings these operations into one equation.

  • The intervention-location study is useful but seems like a minor contribution. It supports a reasonable design choice, but does not provide a deeper explanation of emotional control. The paper does not closely examine why scale-and-shift modulation works, how different emotion transformations interact, or when their combinations fail.

  • The full inference process is more involved than the efficiency claims suggest. Appendix A.4 describes an initial unsteered generation, two controlled generations, and two timestamp-estimation stages using Whisper tiny. The system therefore depends on an external alignment model and repeated generation. It is unclear whether the reported runtime includes all these stages.

  • The comparisons do not clearly establish the gain over the closest prior methods. Related approaches such as EmoSteer-TTS, CoCoEmo, WeSCon, and TED-TTS are discussed but are not directly evaluated in the result tables. Comparisons with general TTS systems adapted through segment-wise generation provide useful context, but do not fully establish the novelty of the proposed control mechanism.

Questions for authors
  • Does the reported word-level runtime include the initial synthesis, both controlled synthesis passes, and both Whisper timestamp-estimation stages?

  • Since ECM and the backbone are trained together, how much of the improvement comes from ECM rather than backbone fine-tuning? Under the same training setup, is scale-and-shift modulation better than simpler conditioning methods, such as concatenating an emotion embedding with the input audio features, as in [EmoCtrl-TTS]?

[EmoCtrl-TTS]: Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech

8 comments
AuthorsOpenReview ↗

Replies to Reviewer KyCv: first part

We thank the reviewer for recognizing the value of our intervention-location study and improvements in emotion control and speech quality. We address your concerns as below:

Question 1 and Weakness 3: Full Inference Pipeline and Runtime Accounting

Thanks for these helpful comments. The reported word-level RTF of 0.137 (Tables 1 and 3) includes the initial unsteered synthesis, the first Whisper-tiny-based timestamp estimation, and the first controlled synthesis. The second timestamp estimation and controlled synthesis are part of an optional refinement step (Eq. 8), which was not employed in our main experiments.

We additionally detail the RTF calculated formulations as below:

RTFsingle=T0+A1+T1D1.\mathrm{RTF}_{\mathrm{single}} = \frac{T_0 + A_1 + T_1}{D_1}.

This is the configuration used in Tables 1 and 3 of the main experiments, including the initial unsteered synthesis and one ASR-based temporal-anchor alignment.

RTFrefined=T0+A1+T1+A2+T2D2.\mathrm{RTF}_{\mathrm{refined}} = \frac{T_0 + A_1 + T_1 + A_2 + T_2}{D_2}.

This corresponds to the optional additional alignment refinement. Here:

  • T0T_0 is the time required for the initial unsteered synthesis.
  • T1T_1 and T2T_2 are the times required for the first and second controlled synthesis passes.
  • A1A_1 and A2A_2 are the times required for the first and second alignments, including ASR inference and temporal-anchor construction.
  • D1D_1 and D2D_2 are the final output audio durations for the single-pass and two-pass controlled-generation configurations, respectively.

The table below also reports speech generation RTF for conventional models as a speed reference.

Table A. Inference Efficiency and Control Performance.

MethodGeneration configurationControl Words SER (%) ↑RTF ↓
CosyVoice2Sentence-level generation—0.856
IndexTTS2Sentence-level generation—0.872
F5-TTSSentence-level generation—0.080
OursDefault sentence-level control, without SAI (Table 2)—0.087
OursDefault word-level control, with SAI (Table 1 and 3)0.6460.137
Ours + second alignmentComplete pipeline with a second alignment0.6510.174

These results clarify that the reported word-level RTF of 0.137 includes the initial unsteered synthesis, ASR alignment, and first controlled generation, rather than only the final synthesis pass. The optional refinement in Eq. (8) slightly improves Control Word SER from 0.646 to 0.651, while increasing RTF from 0.137 to 0.174. Both configurations remain faster than real time and substantially more efficient than CosyVoice2 and IndexTTS2 under the same hardware and evaluation conditions. We will clarify the inference pipeline and RTF calculation in the revised manuscript.

AuthorsOpenReview ↗

Replies to Reviewer KyCv: second part

Question 2: ECM Effectiveness and Conditioning Strategy Comparison

Thanks for this valuable comment. We address these two aspects through additional controlled experiments.

1. ECM vs. Backbone Fine-Tuning. To disentangle the contributions of ECM and backbone fine-tuning, we conduct two additional comparisons: (1) training ECM with the backbone frozen and (2) applying standard emotion-conditioned supervised fine-tuning without ECM (F5-TTS conventional ft). All configurations are evaluated on the same test set.

Table B. Ablation study on ECM and backbone fine-tuning.

MethodWER (%) ↓S-SIM ↑UTMOS ↑SER ↑E-SIM ↑
F5-TTS3.1830.9183.2530.1040.449
F5-TTS conventional ft4.2360.9213.2820.2420.505
Ours freeze backbone4.3860.9153.5300.3820.611
Ours4.6700.9244.4720.8980.903

As shown in Table B, ECM improves emotional expressiveness even with the backbone frozen, increasing SER from 0.104 to 0.382 and E-SIM from 0.449 to 0.611 over F5-TTS. In contrast, standard emotion-conditioned fine-tuning without ECM achieves only 0.242 SER and 0.505 E-SIM. Joint training achieves the best overall performance, further improving SER to 0.898 and E-SIM to 0.903. These results demonstrate that ECM provides benefits beyond backbone fine-tuning, while joint optimization further enhances its effectiveness on emotional expression.

2. Scale-and-Shift vs. Embedding Concatenation. To further examine the effectiveness of scale-and-shift modulation, we implement an EmoCtrl-style [1] conditioning baseline on F5-TTS (F5-TTS-EmoCtrl), which concatenates emotion embeddings with the input audio features. We compare the two conditioning strategies under the same training and evaluation settings.

Table C. Comparison between ECM and emotion embedding concatenation.

MethodWER (%) ↓S-SIM ↑UTMOS ↑SER ↑E-SIM ↑
F5-TTS3.1830.9183.2530.1040.449
F5-TTS-EmoCtrl3.9170.9203.4210.2310.572
Ours4.6700.9244.4720.8980.903

As shown in Table C, ECM improves SER from 0.231 to 0.898 and E-SIM from 0.572 to 0.903 compared with the concatenation-based conditioning baseline. It also achieves higher UTMOS and comparable speaker similarity, although with a slightly higher WER. These results suggest that scale-and-shift modulation may be better suited to our unified multidimensional emotion control setting, rather than indicating a general limitation of concatenation-based conditioning.

We appreciate the insightful conditioning strategy proposed in EmoCtrl-TTS [1], which provides valuable inspiration for emotion-controllable speech synthesis. We will acknowledge and cite this work in the revised manuscript.

[1] Haibin Wu et al. “Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech.” In Proceedings of the 2024 IEEE Spoken Language Technology Workshop (SLT), 2024.

AuthorsOpenReview ↗

Replies to Reviewer KyCv: third part

Weakness 1: Methodological Novelty and Unified Control Contribution

Thanks for this insightful comment. Our contribution lies not in the individual operations themselves, but in how localization, intensity, and composition are learned and coordinated within a unified framework. We clarify this contribution from two perspectives.

1. From a Design Perspective. Existing approaches generally address these control dimensions separately or support only subsets of them. In contrast, ControlAnything investigates how all three dimensions can be enabled through a shared emotion-conditioned learning objective.

Specifically, emotion-category supervision learns reusable emotion-specific transformations, which support continuous intensity adjustment and multi-emotion composition without additional dimension-specific supervision or retraining. To coordinate these transformations with word-level localization, SAI progressively reinforces temporal anchors within the DiT backbone, enabling localized emotional modulation while preserving contextual interactions. Thus, the three dimensions are connected through shared emotion representations and coordinated temporal intervention, rather than independently trained and subsequently assembled.

2. From an Evaluation Perspective. We investigate not only individual control capabilities but also the necessity of the unified design and its compatibility under multidimensional requests. First, the ablation study (Table 8) shows that replacing emotion-specific modulation with a single shared mapping reduces H-Rate from 0.458 to 0.305 and N-MOS from 4.23 to 3.18, while increasing DNSV-Pro from 9.916 to 29.803. Second, our intervention-placement study (Table 1) identifies the trade-off between localization accuracy and boundary continuity, showing that progressive In-DiT anchoring achieves a more balanced performance than input-level or output-level intervention.

Finally, our pairwise and three-dimensional evaluations (Table 6 and Appendix A.2) demonstrate that the proposed framework maintains effective controllability when multiple control dimensions are applied simultaneously, supporting their compatibility within a unified generation process.

Overall, our contribution is not simply to integrate modules, but to demonstrate how reusable emotion-specific transformations and progressive temporal intervention can jointly enable multidimensional emotional control under a shared training objective.

Reviewer_YTzfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ControlAnything combines word-level emotion placement, intensity adjustment, and emotion mixing in a TTS system. SAI uses timestamps from generated speech to restrict intervention to selected words, while ECM applies learned emotion-specific scaling and shifting inside the model. ECM and the backbone are trained together, and word-level generation uses repeated synthesis and alignment to correct timing drift. Experiments cover individual and combined control settings, with the strongest results obtained on F5-TTS, including improvements on CREMA-D and smoother transitions than segment-based synthesis.

Critical strengths

S1: On CREMA-D, the F5-TTS variant raises SER from 0.249 to 0.796 while reducing WER from 3.075% to 2.778%, providing evidence that the learned control transfers beyond the training corpora (Table 2).

S2: Removing SAI raises DNSV-Pro from 9.916 to 21.809 and lowers naturalness MOS from 4.23 to 3.58, supporting its role in maintaining speech quality during local control (Table 8).

Critical weaknesses

W1: The backbone is jointly fine-tuned without a matched emotion-conditioned fine-tuning baseline, and comparisons with WeSCon, EmoSteer-TTS, and CoCoEmo are absent, leaving the source of the gains unclear.

W2: The efficiency claim is difficult to assess because Appendix A.4 requires three synthesis passes and two word-alignment calls, but the reported RTF does not specify whether these steps are included.

Questions for authors

Q1: How much does ECM improve over a standard emotion-conditioned F5-TTS fine-tune using the same data and training budget?

Q2: With joint-control correlations of ρI = 0.326 and ρW = 0.184, what evidence shows that listeners hear the requested mixtures within the selected words without unwanted emotion changes elsewhere?

Q3: Does the word-level RTF of 0.137 include the initial unsteered synthesis, both controlled passes, and both alignment calls on the same hardware used for the baselines?

Q4: How many utterances and listeners were used in each evaluation, and were speakers and transcripts kept disjoint between the training and test sets?

7 comments
AuthorsOpenReview ↗

Replies to Reviewer YTzf: first part

We thank the reviewer for the careful evaluation of our work and the constructive comments on baseline comparisons, the source of ECM's gains, inference efficiency, and the perceptual validity of local emotion control. We also appreciate the questions about evaluation scale and data splits. We address these concerns and questions below.

Question 1: Comparison between ECM and standard fine-tune approach

Thanks for this insightful comment. For F5-TTS, the official implementation does not provide a dedicated emotion-conditioned fine-tuning pipeline. Instead, its officially adopted approach achieves emotion control through reference audio, which we also follow in Table 2 and our other evaluations.

To further assess the effectiveness of ECM over standard emotion-conditioned fine-tuning, we introduce a baseline trained with conventional supervised fine-tuning across all emotion categories, without ECM (F5-TTS conventional ft). For a fair comparison, all experiments are conducted using the same A100 (40 GB) GPU budget and the same training and test datasets as in the main paper.

Table A. Effectiveness of ECM in Emotion-Conditioned Fine-Tuning.

MethodWER (%) ↓S-SIM ↑UTMOS ↑SER ↑E-SIM ↑
F5-TTS3.1830.9183.2530.1040.449
F5-TTS (conventional ft)4.2360.9213.2820.2420.505
Ours4.6700.9244.4720.8980.903

Compared with standard emotion-conditioned fine-tuning, ECM substantially improves SER from 0.242 to 0.898 and E-SIM from 0.505 to 0.903, while also improving UTMOS and speaker similarity. These results indicate that the gains in emotional expressiveness cannot be attributed solely to backbone fine-tuning, supporting the effectiveness of ECM in learning emotion-specific modulation.

Question 2: Emotion Mixtures and Preservation Within and Outside Target Words

Thanks for this valuable comment. In addition to the correlations of ρI = 0.326 and ρW = 0.184 in Table 4, 5, and 6 report human perceptual ratings for word-level control, intensity control, and joint control across all three dimensions.

To further examine listeners' perceptions within and outside the selected regions, we designed an additional subjective evaluation under joint-control conditions, combining regional E-MOS with sentence-level N-MOS. We first designed the relevant auditory metric:

  • E-MOS within Target Words (E-MOS-TW): Rates how well the emotion within target words matches the specified emotion (1–5).
  • Emotion Preservation Outside Target Words (EPO-TW): Rates whether non-target words preserve their original emotional expression compared with the unsteered reference (1–5; higher is better).
  • Sentence-level N-MOS (N-MOS): Rates the naturalness of the complete synthesized sentence (1–5).
  • Emotion-mixture Identification Rate (EIR): Percentage of evaluations in which listeners correctly identify both target emotions without selecting additional emotions, before seeing the target labels.

The experiment results as below:

Table B. Perceptual Evaluation of Joint Emotion Control.

MethodE-MOS-TW ↑EPO-TW ↑N-MOS ↑EIR (%) ↑
F5-TTS-seg3.72±0.414.283.36±0.3771.5
Ours w/o SAI4.18±0.362.763.71±0.4381.3
Ours4.46±0.194.624.43±0.3291.7

The above MOS ratings follow the same protocol as in the main paper (40 human expert on 300 samples each).

As shown in the table, removing SAI leaves the E-MOS within target words at 4.18 but reduces the emotion-preservation score outside them to 2.76, indicating the role of local anchoring in limiting emotional spillover. These results support listeners' ability to identify the requested emotion mixture within the selected words while emotional expression elsewhere and overall sentence naturalness are preserved.

AuthorsOpenReview ↗

Replies to Reviewer YTzf: second part

Question 3 and Weakness 2: Full-Pipeline Inference Cost and RTF Accounting

Thanks for these helpful comments. The reported word-level RTF of 0.137 includes the initial unsteered synthesis, one ASR-based alignment, and the first controlled synthesis. The second alignment and controlled synthesis in Eq. (8) constitute an optional refinement step that was not employed in our main experiments and is therefore excluded from the reported RTF.

We additionally detail the RTF calculated formulations as below:

RTFsingle=T0+A1+T1D1.\mathrm{RTF}_{\mathrm{single}} = \frac{T_0 + A_1 + T_1}{D_1}.

This is the configuration used in Tables 1 and 3 of the main experiments, including the initial unsteered synthesis and one ASR-based temporal-anchor alignment.

RTFrefined=T0+A1+T1+A2+T2D2.\mathrm{RTF}_{\mathrm{refined}} = \frac{T_0 + A_1 + T_1 + A_2 + T_2}{D_2}.

This corresponds to the optional additional alignment refinement. Here:

  • T0T_0 is the time required for the initial unsteered synthesis.
  • T1T_1 and T2T_2 are the times required for the first and second controlled synthesis passes.
  • A1A_1 and A2A_2 are the times required for the first and second alignments, including ASR inference and temporal-anchor construction.
  • D1D_1 and D2D_2 are the final output audio durations for the single-pass and two-pass controlled-generation configurations, respectively.

The table below also reports speech generation RTF for conventional models as a speed reference.

Table C. Inference Efficiency and Control Performance.

MethodGeneration configurationControl Words SER (%) ↑RTF ↓
CosyVoice2Sentence-level generation—0.856
IndexTTS2Sentence-level generation—0.872
F5-TTSSentence-level generation—0.080
OursDefault sentence-level control, without SAI (Table 2)—0.087
OursDefault word-level control, with SAI (Table 1 and 3)0.6460.137
Ours + second alignmentComplete pipeline with a second alignment0.6510.174

These results clarify that the reported word-level RTF of 0.137 includes the initial unsteered synthesis, ASR alignment, and first controlled generation, rather than only the final synthesis pass. The optional refinement in Eq. (8) slightly improves Control Word SER from 0.646 to 0.651, while increasing RTF from 0.137 to 0.174. Both configurations remain faster than real time and substantially more efficient than CosyVoice2 and IndexTTS2 under the same hardware and evaluation conditions. We will clarify the inference pipeline and RTF calculation in the revised manuscript.

Question 4: Evaluation Sample Sizes, Speaker and Transcript Separation

Thanks for this edtailed comment.

1. Evaluation Sample Sizes: Our listening study involved 40 evaluators, comprising Ph.D. holders and senior Ph.D. students specializing in artificial intelligence, all with sufficient English listening proficiency. Each evaluator assessed 300 audio samples for each evaluation dimension requiring MOS ratings. The average completion time was approximately 50 minutes per evaluator, with compensation provided at a rate of 30 units per hour in local currency.

2. Speaker and Transcript Separation: For OOD evaluation (Lines 280–281), we use CREMA-D, which is entirely excluded from our training corpus, to assess generalization to unseen speakers and linguistic content.

For ID evaluation, existing emotional speech datasets do not readily support strict speaker- and transcript-disjoint splits while covering all seven emotion categories. Therefore, following the data partitioning practices of ESD [1] and WeSCon [2], we seek to minimize overlap by avoiding identical speaker–transcript pairs between training and test sets wherever possible.

[1] Kun Zhou et al. Seen and Unseen Emotional Style Transfer for Voice Conversion with a New Emotional Speech Dataset. ICASSP, 2021.

[2] Tianrui Wang et al. Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis. NeurIPS, 2025.

AuthorsOpenReview ↗

Replies to Reviewer YTzf: third part

Weakness 1: Baseline Comparisons and the Source of Performance Gains

For EmoSteer-TTS, CoCoEmo, WeSCon, and TED-TTS, the official releases do not provide the weights and training codes needed for direct reproduction.

We emailed the corresponding authors of the models mentioned above, but ultimately only obtained partial code resources and technical assistance. Based on these materials and the reproducibility details provided in the respective papers, we conducted additional evaluations.

1. Sentence-Level Emotion Control. All four methods support sentence-level emotional speech synthesis. We therefore first compare their overall emotional expressiveness.

Table D. Sentence-level emotion control.

MethodWER (%) ↓S-SIM ↑UTMOS ↑SER ↑E-SIM ↑
EmoSteer-TTS4.2910.6444.2530.7600.721
CoCoEmo4.5260.9014.1580.6740.692
WeSCon5.3940.9014.2310.7500.747
TED-TTS3.9920.9204.2570.7330.766
Ours4.6700.9244.2950.8980.895

Our method achieves the highest SER and E-SIM, outperforming the best competing results by 0.138 and 0.129, respectively, while maintaining competitive speech quality and speaker similarity.

2. Word-Level Emotion Control. For word-level emotion control, only the WeSCon and TED-TTS methods support word-level emotion injection.

Table E. Word-level emotion control.

MethodWER (%) ↓DNSV-Pro ↓S-SIM ↑AutoPCP ↑Emo2v. ↑Aro. ↑
WeSCon4.3725.8140.8972.7140.5890.656
TED-TTS3.4766.7520.8912.6890.6040.689
Ours3.5625.0740.8832.8270.6460.733

Our method reduces DNSV-Pro by 12.7% and improves AutoPCP by 0.113 over WeSCon, while achieving the highest Emo2v. and Aro. scores. These results demonstrate improved localized emotion control and prosodic consistency.

3. Emotion Intensity Control. For emotion intensity control, only EmoSteer-TTS and CoCoEmo support this dimension. We therefore compare our method with these two approaches.

Table F. Emotion intensity control.

MethodρI ↑ΔEmo2v. ↑WER (%) ↓S-SIM ↑
EmoSteer-TTS0.5310.4413.9280.920
CoCoEmo0.5220.4564.0050.904
Ours0.5460.4744.2360.918

As shown in Table F, our method improves ρI and ΔEmo2v. by 0.015 and 0.018 over the best competing results, respectively, indicating more consistent intensity control and a wider modulation range while maintaining comparable content accuracy and speaker similarity.

4. Multi-Emotion Composition. For multi-emotion composition, we consider two evaluation settings: (A) same-span emotion mixing and (B) within-sentence emotion switching. EmoSteer-TTS and CoCoEmo support the former, while WeSCon and TED-TTS primarily support the latter.

Table G-A. Emotion composition: same-span emotion mixing.

MethodE-SIM ↑ρW ↑H-Rate ↑WER (%) ↓S-SIM ↑
EmoSteer-TTS0.8040.5520.3581.3220.925
CoCoEmo0.8400.6360.4301.0410.937
Ours (F5-TTS)0.8610.6620.4580.7090.924

For same-span emotion mixing, our method improves E-SIM and ρW by 0.021 and 0.026 over CoCoEmo, respectively, and H-Rate by 2.8 percentage points, while achieving the lowest WER. These results indicate more accurate emotion mixtures and better adherence to composition weights.

Table G-B. Emotion composition: within-sentence emotion switching.

MethodE-SIM ↑ρW ↑H-Rate ↑WER (%) ↓S-SIM ↑
WeSCon0.8610.7760.8630.9450.978
TED-TTS0.8750.7960.8710.8020.981
Ours0.8910.8220.9080.6210.987

For within-sentence emotion switching, our method improves E-SIM and ρW by 0.016 and 0.026 over TED-TTS, respectively, while achieving the lowest WER and highest speaker similarity. These results indicate more accurate emotion transitions with better content preservation.

5. Matched Fine-Tuning Baseline. As detailed in our response to Q1, we additionally evaluated emotion-conditioned fine-tuning without ECM under matched training data and computational budgets. ECM improves SER from 0.242 to 0.898 and E-SIM from 0.505 to 0.903, confirming gains beyond backbone fine-tuning.

Overall, these additional experiments demonstrate the effectiveness of our unified framework across word-level localization, intensity control, and emotion composition. We will incorporate these comparisons and implementation details into the revised manuscript.

[1] Tianrui Wang et al. Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis. NeurIPS, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Additional Experiments and Clarifications

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful and constructive feedback. Below, we summarize the results made in response.

Unified Control and ECM’s Contribution. We clarified how SAI anchors emotional changes to target words and how ECM reuses the same learned transformations to adjust intensity and combine emotions under a shared training objective. To distinguish ECM’s contribution from backbone fine-tuning, we added a frozen-backbone baseline and a matched fine-tuning baseline without ECM. We also compared ECM’s scale-and-shift modulation with a simpler approach that concatenates emotion embeddings with input features.

Comparisons with More Baselines. We added more comparisons with EmoSteer-TTS, CoCoEmo, WeSCon, and TED-TTS on the control dimensions each method supports. For word-level control, we clarified how the segment-based baselines join separately synthesized segments and included WeSCon and TED-TTS as baselines with native word-level control.

Inference Cost and Alignment Accuracy. We clarified that the reported word-level real-time factor (RTF) includes an initial synthesis pass, one ASR alignment, and one controlled synthesis pass. We reported the additional runtime and localization benefit of a second, optional alignment and synthesis pass. We also perturbed word timestamps to measure SAI’s sensitivity to boundary errors.

Perceptual Evaluation and Failure Cases. We added listening tests to assess whether listeners hear the requested emotion mixture in the target words, whether the surrounding words preserve their original emotional expression, and whether the sentence sounds natural overall. We also examined contrasting emotions across separate spans and conflicting emotions within the same span, discussing cases where one emotion may dominate or the two may neutralize each other.

Evaluation Scale and Data Splits. We documented the listening-study protocol and the numbers of utterances and listeners. We also clarified how the in-distribution splits minimize overlap in speaker–transcript pairs and confirmed that CREMA-D was held out entirely for out-of-distribution evaluation.

We would like to highlight a positive development during the normal discussion period:

ReviewerInitial RatingAfter DiscussionUpdate DateOfficial Comment by Reviewer
Reviewer YTzf2218 Nov 2026Thank you for the detailed response and the considerable effort devoted to the additional experiments. I appreciate the expanded comparisons, perceptual evaluation, and clarifications of inference cost and evaluation protocols. These additions provide useful context for the concerns raised in my review
Reviewer 6jCh3417 Nov 2026-

We have incorporated these additional results and clarifications into the revised manuscript, with the new material highlighted in blue\color{blue}{\text{blue}}. We sincerely thank the reviewers for their constructive feedback, which has helped strengthen the empirical evidence and clarity of our work. We hope this summary will assist the Area Chair in assessing our revised submission.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.