- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents an emotion control framework for non-autoregressive speech synthesis. It supports three types of control: specifying which words carry emotions, how to adjust the intensity of emotions, and mixing multiple emotions within the same segment. Specifically, SAI maps the target words to corresponding time frames, limiting the location of emotional intervention. ECM controls the intensity and mixing ratio of emotions by scaling and combining the modulation parameters of different emotions. The results show that the proposed method can achieve local emotion control while making the transitions between segments more natural.
Critical strengths
- The three types of emotion control can be used in combination. ControlAnything not only can specify which words carry emotions, but also can simultaneously adjust the overall emotional intensity and the relative weights of various emotions on these words.
- The experiment is thorough. The authors not only evaluated the three control capabilities separately, but also tested the performance when all three were enabled simultaneously, and verified the roles of SAI and ECM through module ablation.
Critical weaknesses
- The method innovation of ECM is rather limited. ECM learns a set of scaling and offset parameters for each emotion, then controls the intensity by modulating the amplitude through scaling, and achieves emotion blending through weighted superposition. The existing ablation studies mainly demonstrate the usefulness of ECM within this framework, but they are not sufficient to show its advantages over more direct feature modulation schemes.
- The computational cost of word-level inference is not yet clear. The complete process includes one synthesis without emotion intervention, word-level alignment based on ASR, and two rounds of controlled synthesis. The paper does not clearly state whether the reported RTF takes into account these steps, making it difficult to fairly compare the inference efficiency with other methods.
Questions for authors
- Could you please clarify whether the RTF in Table 3 is calculated based on the entire word-level control process, or only accounts for the time consumption of the synthesized part?
- When conducting word-level comparison, how do you concatenate the generated baseline speech segments? Is it done by direct concatenation, or is some smoothing processing such as cross-damping applied?
- If ASR overestimates the start and end times of words, to what extent can the second alignment correct this deviation? Have you encountered situations where the emotion of the target word is not properly adjusted, or where adjacent words are also affected?
5 comments
Replies to Reviewer 6jCh: second part
Question 2: Fairness of Comparisons with Segment-Wise Baselines
Thanks for this valuable comment. We clarify this issue from two perspectives.
1. Segment-wise Baseline Settings. Currently, no publicly accessible TTS models support native word-level emotion control. Therefore, we followed the segment-wise evaluation setting of WeSCon [1] for baselines such as F5-TTS-seg and IndexTTS2-seg. This setting approximates word-level control through segment-wise synthesis and concatenation.
We acknowledge that segment-wise synthesis may introduce boundary discontinuities, potentially disadvantaging these baselines in transition smoothness evaluations.
2. Native Word-level Comparisons. To address this concern, we contacted the authors of WeSCon and TED-TTS and just obtained partial code resources and technical assistance. Based on these materials and the implementation details provided in their papers, we reproduced both methods and conducted additional evaluations using the same training and test data as our method.
Table B. Word-level emotion control.
| Method | WER (%) ↓ | DNSV-Pro ↓ | S-SIM ↑ | AutoPCP ↑ | Emo2v. ↑ | Aro. ↑ |
|---|---|---|---|---|---|---|
| WeSCon | 4.372 | 5.814 | 0.897 | 2.714 | 0.589 | 0.656 |
| TED-TTS | 3.476 | 6.752 | 0.891 | 2.689 | 0.604 | 0.689 |
| Ours | 3.562 | 5.074 | 0.883 | 2.827 | 0.646 | 0.733 |
As shown in Table B, our method achieves the lowest DNSV-Pro of 5.074 and the highest AutoPCP, Emo2v., and Aro. scores of 2.827, 0.646, and 0.733, respectively. These results demonstrate improved prosodic consistency and localized emotion control over native word-level methods, further supporting the effectiveness of our approach beyond comparisons with segment-wise baselines.
[1] Tianrui Wang et al. Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis. NeurIPS, 2025.
Replies to Reviewer 6jCh: third part
Question 3: SAI Robustness to Boundary Perturbations
Thanks for this helpful comment. We conducted an additional timestamp perturbation experiment to evaluate the robustness of SAI. We randomly perturbed the word boundaries across the entire test set, using perturbation ranges of ±20, ±50, and ±100 ms. We report the residual localization error rate, ER@50 ms: the interval in which control is actually applied is compared with independent manual annotations of the target-segment boundaries in the corresponding final output. A request is counted as an error if either the start or end boundary deviates by more than 50 ms. Missing target words and localization failures are also counted as errors. The results are shown below.
Table C. Robustness to ASR Boundary Perturbations.
| Initial boundary perturbation | Single-pass ER@50 ms ↓ | With refinement ER@50 ms ↓ | Single-pass Emo2v. ↑ | With refinement Emo2v. ↑ |
|---|---|---|---|---|
| No additional perturbation | 3.0% | 2.0% | 0.646 | 0.650 |
| ±20 ms | 3.2% | 2.1% | 0.643 | 0.649 |
| ±50 ms | 4.7% | 2.5% | 0.631 | 0.646 |
| ±100 ms | 12.6% | 3.0% | 0.601 | 0.638 |
With the original ASR timestamps, the default single-pass configuration already achieves a low localization error rate of 3.0%, which decreases further to 2.0% with optional refinement. As the perturbation magnitude increases, localization errors in the single-pass configuration gradually increase, while two-pass refinement reduces residual misalignment and better preserves the target emotional expression. These results indicate that SAI can tolerate small errors in the initial boundaries. When the initial localization error is larger or an application requires higher temporal precision, two-pass refinement can further reduce localization errors.
Weakness 1: ECM Design and Methodological Novelty
Thanks for this insightful comment. The main contribution of ECM lies in learning reusable emotion-specific modulation that support multiple control dimensions under a shared training objective. We clarify two key aspects below.
1. Unified Emotion Learning. Rather than introducing separate training objectives for intensity and composition, ECM learns distinct emotion-specific transformations using the backbone's original conditional flow-matching objective with emotion-category supervision. These learned transformations can subsequently be scaled for continuous intensity adjustment and combined for multi-emotion composition, without requiring intensity-specific annotations, mixed-emotion training targets, or dimension-specific retraining. Thus, the key contribution is not the scaling or weighted combination itself, but the ability to derive multiple control capabilities from a common set of learned emotion-specific transformations.
2. Effectiveness of Emotion-Specific Modulation. Our ablation study (Table 8) further supports this design. Replacing emotion-specific modulation with a single shared mapping for all emotions reduces H-Rate from 0.458 to 0.305 and N-MOS from 4.23 to 3.18, while increasing DNSV-Pro from 9.916 to 29.803. These results demonstrate the importance of learning distinct emotion-specific transformations rather than relying on a single undifferentiated mapping.
Overall, ECM provides a unified emotion-conditioning mechanism that supports categorical emotion expression, continuous intensity adjustment, and multi-emotion composition without separate training objectives for each control dimension.
Replies to Reviewer 6jCh: first part
We thank the reviewer for recognizing our unified framework for fine-grained emotional control and the comprehensive evaluation supporting its effectiveness. We address the reviewer's concerns below.
Question 1 and Weakness 2: Full-Pipeline Runtime for Word-Level Control
Thanks for these helpful comments. The reported word-level RTF of 0.137 includes the initial unsteered synthesis, one ASR-based alignment, and the first controlled synthesis. The second alignment and controlled synthesis in Eq. (8) constitute an optional refinement step that was not employed in our main experiments and is therefore excluded from the reported RTF.
We detail the RTF calculated formulations as below:
RTFsingle=D1T0+A1+T1.
This is the configuration used in Tables 1 and 3 of the main experiments, including the initial unsteered synthesis and one ASR-based temporal-anchor alignment.
RTFrefined=D2T0+A1+T1+A2+T2.
This corresponds to the optional additional alignment refinement. Here:
The table below also reports speech generation RTF for conventional models as a speed reference.
Table A. Inference Efficiency and Control Performance.
These results clarify that the reported word-level RTF of 0.137 includes the initial unsteered synthesis, ASR alignment, and first controlled generation, rather than only the final synthesis pass. The optional refinement in Eq. (8) slightly improves Control Word SER from 0.646 to 0.651, while increasing RTF from 0.137 to 0.174. Both configurations remain faster than real time and substantially more efficient than CosyVoice2 and IndexTTS2 under the same hardware and evaluation conditions. We will clarify the inference pipeline and RTF calculation in the revised manuscript.