acceptodds
Under review as a conference paper at ICLR 2027

EmoScale: Localized Emotion Intensity Control for Speech Synthesis via Twin-Forward Flow Matching

Abstract

Recent flow-matching and diffusion-transformer (DiT)-based text-to-speech (TTS) models achieve high naturalness, yet precise and localized control of emotional expression remains challenging. Existing instruction- or encoder-conditioned methods inject emotion through labels, embeddings, or prompts, while training-free steering methods extract emotion directions post hoc without explicitly enforcing disentanglement. These approaches provide limited control over the intensity and temporal extent of emotional expression, as the realised emotion can be influenced by the model’s internal representations and interactions between emotion conditioning and linguistic content. Moreover, conventional categorical emotion representations and natural-language prompts provide limited means to precisely specify how strongly and when an emotion should be expressed. To address these challenges, we propose EmoScale, a training paradigm that explicitly disentangles the velocity field of DiT-based flow-matching model into semantic and emotion components without requiring an external emotion embedding. First, we introduce a disentanglement regularizer that encourages the emotion velocity to be orthogonal to the semantic velocity. We further show that the resulting emotion representation enables continuous control of emotional intensity through linear interpolation of its corresponding acoustic attributes, including F0, periodicity and envelope, followed by speech reconstruction. This allows us to adjust the intensity of an emotional speech while preserving perceptual naturalness. We evaluate EmoScale against encoder-conditioned and training-free steering baselines in terms of naturalness, emotion recognition accuracy, content preservation, and localized emotion control. Results demonstrate that explicitly disentangling the velocity field enables precise and interpretable control over emotional intensity without requiring additional trainable auxiliary modules. The resulting RA scores exhibit a clear and consistent trend as the emotion intensity is varied, confirming the effectiveness of the proposed control mechanism.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio generation, emotion, diffusion, interpretability

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_xSkkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is different from previous global emotion control. It considers a novel localized emotion intensity control. In the scenario, it supports the emotion of the entire sentence while also allowing the adjustment of the emotion of the latter half of the sentence based on the TTS architecture. The generation velocity field was divided into two parts, one without emotion and one rich in emotion. And these two parts are orthogonal to each other. Experiments on various datasets show that the method can evoke a clearly perceptible strong emotion.

Critical strengths
  1. The disentanglement of velocity field is simple but effective. It can adjust the emotional intensity of the speech without significantly altering the content of the speech.
  2. The emotional intensity on the OOD data changes in accordance with the variations of the control parameters, which proves the generalization performance of EmoScale.
Critical weaknesses

The current experiments have not yet clarified the contribution of DiT velocity field decomposition to the control of emotional intensity. The final intensity regulation relies on acoustic interpolation based on WORLD. Therefore, without comparing the results with and without velocity field decomposition under the same WORLD interpolation settings, it is difficult to determine exactly how much additional benefit the core design of the paper brings to local intensity control.

Questions for authors
  1. How do you determine that the two velocity components indeed correspond to the language content and the emotion respectively? I understand that the orthogonal constraint can limit the geometric relationship between the two components, but even if they are orthogonal, each of them may still carry some semantic and emotion information. Besides the orthogonal constraint, what other measures are there in the training process that make the two components play different roles?
  2. Could you specifically explain the synthesis process from the DiT output to the final waveform? I did not fully align with the position of WORLD in Section 3 and Figure 1. What I particularly want to understand is whether the output of DiT already shows perceptible changes in emotional intensity before the WORLD interpolation is performed. If you have a comparison before and after interpolation, could you explain what roles the velocity decomposition and WORLD interpolation respectively play?
1 comment
AuthorsOpenReview ↗

Response to Reviewer xSkk

Thanks to you the reviewer to spend efforts to provide the suggestions. We have respond below point by point to clarify potential misunderstandings.

W1. Effect of orthogonality

EmoScale decomposes the flow-matching velocity field itself into a semantic and an emotion component, learned jointly from paired data and kept frame-synchronous, so that emotion becomes an additive, per-frame quantity that can be scaled globally or locally. the velocity decomposition provides frame-synchronous, interpretable emotion and semantic components, while WORLD is used only to remove the comb-like artefacts of direct mixing. Replacing our endpoints with independently generated ones under the same WORLD interpolation yields a flat intensity axis and breaks the content at intermediate α, which isolates the contribution of the decomposition.

To verify this, we replaced our endpoints with neutral and emotional versions generated independently by an existing emotional TTS model (CosyVoice3-instruct, with the real neutral recording as the speaker prompt), aligned them by DTW, and applied exactly the same pitch-aligned interpolation (Table below). Independently generated endpoints do not provide a clean emotional direction: the generated emotional end is not more emotional than the neutral end, the intensity axis is flat. They are also not time-synchronous (duration ratio 1.55), so the content breaks down at intermediate α (CER 0.26 at α = 0.5), and segmented α does not move the intensity in the intended direction. EmoScale, in contrast, yields a monotonic intensity axis and correct localised control.

MetricCosyVoice3 endpointsEmoScale (ours)
RA of the emotional end (neutral end)−0.17 (−0.06)0.69 (−0.03)
Normalised RA at α = 0 / 0.5 / 1−0.43 / −0.50 / −0.55−0.11 / 0.33 / 0.95
CER at α = 0.50.260.18
Segmented α (0 → 1): RA of 1st / 2nd half0.15 / −0.510.25 / 0.57

Endpoint-source ablation on the RAVDESS utterances of Table 3, using identical pitch-aligned interpolation.

W2. Limited training scalability

We view data efficiency as a strength of the framework rather than a limitation. EmoScale is trained on only neutral–emotional pairs from ESD, yet it generalises to three out-of-domain datasets (RAVDESS, CREMA-D and IEMOCAP) with different speakers, recording conditions and emotion distributions (Table 1). The current scale is therefore sufficient for the task, and scaling to large in-the-wild corpora is beyond the scope of this paper.

Q1. Disentanglement verification

Yes. In EmoScale, orthogonality is the constraint that realises the semantic–emotion separation in training, and the separation is also evidenced empirically. (i) Figure 2 shows that the emotion velocities cluster by emotion rather than by speaker or sentence, while the semantic velocities remain close to each other. (ii) Generating with the semantic component alone yields natural neutral speech, and adding the emotion component yields emotional speech whose intensity increases with α (Table 1, Figure 3).

Q2. Clarification on restruction process

The final waveform is produced by the vocoder, as shown in Figure 1. The “waveform” at Line 268 refers to an intermediate step: WORLD resynthesises the two endpoints (acquired by flow-matching with two velocity) with a shared F0 contour; the resynthesised signals are then converted back to mel-spectrograms and mixed according to α, and the mixed mel-spectrogram is passed to the vocoder to produce the final audio. We will make this clear in Section 3 in camera-ready version.

In conclusion, we sincerely thank the reviewer again for their constructive feedback and thoughtful consideration. We hope that our clarifications have satisfactorily addressed the concerns raised and resolved the misunderstandings. If these clarifications have changed the reviewer's initial assessment, we would greatly appreciate your consideration of an upward revision of the overall score.

Reviewer_kssyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • This paper proposes EmoScale, a framework for localized emotion intensity control in speech synthesis. The key idea is to decompose the velocity field of a flow-matching model into a neutral speech component and an emotion residual component, using two DiT networks with an orthogonality regularizer.

  • For emotion intensity control, the trained model generates both neutral and emotional speech, and then interpolates their acoustic features using WORLD, an existing speech analysis and synthesis system. By adjusting the interpolation coefficient, the method enables continuous and temporally localized emotion intensity control.

Critical strengths
  • Meaningful problem setting. Fine-grained emotion intensity control is an important problem in expressive speech synthesis, particularly when emotion needs to be adjusted within specific temporal regions.

  • Intuitive formulation. Decomposing the velocity field into neutral and emotion-residual components provides a conceptually straightforward approach to modeling emotional variations in speech.

  • Evaluation across multiple datasets. The authors evaluate the method on five datasets, including out-of-domain datasets, and report both objective and subjective evaluations. The results show a consistent monotonic relationship between the interpolation coefficient and the measured emotion intensity.

Critical weaknesses

W1. Noticeable artifacts, degraded speech naturalness, and inconsistent emotion controllability

After listening to the provided audio samples, I observed noticeable artifacts and degradation in speech naturalness across multiple examples.

For instance:

Sad (Chinese): "带女友出去好好吃上一顿。" contains noticeable artifacts in the segmented-α result.

ESD-English: "How suspicious this fox." exhibits audible artifacts, particularly around "fox".

Sad: "The end part is fish mouth." contains noticeable artifacts toward the end of the utterance.

Several generated samples exhibit artifacts that are absent from their corresponding neutral references, raising concerns about pronunciation fidelity and content preservation.

Furthermore, emotion intensity control appears inconsistent across samples. For example, in the ESD-English sample "How suspicious this fox.", the perceptual differences across α values are relatively subtle and difficult to distinguish.

Overall, the qualitative results do not fully support the claimed preservation of speech naturalness and reliable emotion intensity controllability.

W2. Limited methodological novelty and insufficient justification for post-hoc WORLD interpolation

Although the paper proposes semantic-emotion disentanglement through twin flow-matching networks, the actual emotion intensity control is achieved through a heuristic, post-hoc interpolation of WORLD acoustic features extracted from fully generated neutral and emotional speech.

It is unclear whether the proposed velocity decomposition is essential for achieving such controllability. In principle, a similar interpolation strategy could be applied to neutral and emotional speech generated by existing emotion-controllable TTS models.

Furthermore, since the model explicitly learns an emotion-residual velocity field, direct modulation of this component during flow-matching integration would be a more natural use of the proposed decomposition. Instead, the method requires generating two complete speech candidates before performing acoustic interpolation.

This design appears computationally inefficient and leaves the benefit of the proposed velocity decomposition insufficiently justified.

More broadly, the paper does not clearly establish how the proposed decomposition differs from existing residual-based or disentangled flow-matching approaches, beyond its specific application to emotional speech synthesis.

W3. Insufficient ablation studies to justify the proposed Twin-DiT and conditioning architecture

The proposed architecture employs two independently trainable DiTs, each conditioned on separate adapter outputs, μ_A and μ_Δ. However, the paper does not sufficiently investigate whether these components are necessary for the claimed semantic-emotion disentanglement.

In particular, there is no ablation examining the contribution of the individual conditioning signals, or whether using the same conditioning representation for both branches would yield comparable results.

Moreover, the two DiTs contain approximately 666M trainable parameters in total. Without a comparison against a simpler single-DiT architecture with different neutral/emotional conditions, it is difficult to determine whether the additional architectural complexity is justified.

Although Table 4 investigates the orthogonality coefficient, the model still exhibits substantial emotion intensity controllability without the orthogonality regularizer. This further raises questions about which components are responsible for the observed performance.

W4. Insufficient analysis of the time-dependent velocity fields

The paper lacks a detailed analysis of how the semantic and emotion velocity fields behave across flow-matching timesteps.

According to Section 3, both branches share a noised state Y_t derived from emotional speech, while the neutral branch is supervised with a velocity target corresponding to neutral speech.

This raises a question about the consistency between the training state and the target velocity for neutral generation. In particular, at later timesteps, when the shared state approaches the emotional speech, predicting a velocity corresponding to neutral generation may become challenging.

A timestep-wise analysis of the prediction errors, velocity magnitudes, and relationship between the two velocity fields would help clarify how the intended decomposition is learned throughout the generative process.

W5. Insufficient quantitative evaluation of localized emotion control

Localized emotion intensity control is a central claim of the paper. However, the experimental validation is largely limited to a few qualitative demonstrations and mel-spectrogram visualizations.

There is no systematic quantitative evaluation of temporal localization accuracy, preservation of unaffected regions, or perceptual naturalness at emotion transition boundaries.

This limitation is particularly concerning given the audible artifacts observed in some segmented-intensity samples.

W6. Limited training scalability due to paired-data requirements

The proposed training framework relies on paired neutral and emotional recordings with identical linguistic content and speaker identity, followed by explicit temporal alignment.

Such paired recordings can be expensive to collect and may not be readily available for large-scale or in-the-wild emotional speech datasets.

Although the authors evaluate generalization on out-of-domain datasets, the reliance on paired data during training remains a practical limitation of the proposed framework.

Minor Weakness

Presentation quality of the architecture overview

Figure 1 could be improved, particularly in terms of arrow organization and the relationships between the conditioning modules, the two velocity branches, and the subsequent inference pipeline.

Questions for authors

Q1. Contribution of Twin-Flow Disentanglement

What is the specific contribution of the proposed twin-flow decomposition to emotion intensity control, beyond the WORLD-based acoustic feature interpolation?

Could the authors compare against a simpler baseline where neutral and emotional speech are independently generated using an existing model, followed by the same WORLD interpolation procedure?

Q2. Direct Emotion Velocity Scaling

Since the model explicitly learns an emotion-residual velocity field v_Δ, why is emotion intensity not controlled directly during flow-matching integration, for example through v = v_A + αv_Δ?

A comparison between direct velocity scaling and WORLD-based interpolation would help clarify the necessity and advantages of the proposed inference strategy.

Q3. Necessity of Separate Conditioning Signals and Twin DiTs

How essential are the separate conditioning signals μ_A and μ_Δ for semantic-emotion disentanglement?

It would be valuable to investigate the following ablations:

Removing μ_A or μ_Δ.

Using a shared conditioning representation for both DiTs.

Replacing the two independently trainable DiTs with a single DiT conditioned on neutral/emotional information.

These comparisons would help identify which components contribute most to the claimed disentanglement and controllability.

Q4. Time-dependent Behavior and Training Consistency

Could the authors clarify how the shared noised state Y_t is constructed for the neutral and emotional branches?

In particular, given that Y_t is described as being derived from emotional speech, how is the neutral velocity target made consistent with the underlying flow-matching trajectory?

Additionally, could the authors provide a timestep-wise analysis of the neutral and emotional velocity prediction errors, particularly at later timesteps where the state approaches emotional speech?

Q5. Evaluation of Localized Emotion Control

Could the authors provide a more systematic evaluation of localized emotion control, beyond the presented spectrogram examples?

In particular, how accurately can the method modify emotion intensity within a target temporal region while preserving the acoustic characteristics of the surrounding regions?

A subjective evaluation of transition naturalness and an objective assessment of temporal localization would strengthen this claim.

Q6. Speech Quality and Content Preservation

How do the authors explain the audible artifacts and degradation in speech naturalness observed in several generated samples, particularly in the segmented-α examples?

Could the authors provide further analysis of the sources of these artifacts, including whether they originate from the twin-flow generation process, WORLD-based interpolation, or waveform reconstruction?

Q7. Novelty Relative to Existing Flow-Matching Approaches

Could the authors more clearly distinguish the proposed velocity decomposition from existing residual-based emotion control and disentangled flow-matching methods?

In particular, what is the methodological contribution of enforcing orthogonality between semantic and emotional velocity fields, given that substantial intensity controllability is already observed without the orthogonality regularizer?

3 comments
AuthorsOpenReview ↗

Response to Reviewer kssy

Thanks to you the reviewer to spend efforts to provide the suggestions. We have respond below point by point to clarify potential misunderstandings.

W1. speech quality and emotion control results

We carefully listened to the three samples together with the corresponding real recordings and the baseline outputs. The perceived characteristics are similar across them, so we do not consider them artifacts introduced by EmoScale; we believe they originate from the source recordings. Note that EmoScale takes the emotional recording as input, so its outputs should be compared with the emotional source recording rather than with the neutral reference, which is a different recording. Regarding content degradation, together with what we claim before about the limitation of CER, we claim UTMOS related quality automatic metrics have limitation as well, so its absolute values should be interpreted with care. Emotion itself lowers UTMOS. On real recordings (Table 1), UTMOS drops from neutral to emotional speech: ESD-zh 3.68 → 2.74, ESD-en 4.19 → 3.65, RAVDESS 4.08 → 3.32.

UTMOS under-scores real speech. UTMOS is calibrated on English synthetic speech; a recent study reports median scores of only 2.72 and 2.43 on clean read Greek and English recordings, respectively [arXiv:2609.13498].

Since EmoScale outputs emotional speech by design, part of the gap reflects this metric bias rather than audible degradation. In our listening, outputs above a moderate UTMOS level are perceptually on par with the baselines. Many previous speech generation works have also stated such metrics are not very reliable such as https://arxiv.org/pdf/2203.11389. We stated here again that this may reflect a limitation of current automatic quality metrics rather than a drop in generation quality, and therefore kindly refer the reviewer to hear the samples at https://anonymous.4open.science/r/emoscale-9A11/samples/index.html. Baseline samples generated under identical conditions are now included.

W2. Novelty

We want to claim that our method is novel enough as we provide experimental proof that the emotion and semantic information could be disentangled with explainability and provide practical usage for localised emotion intensity control. Specifically, EmoScale decomposes the flow-matching velocity field itself into a semantic and an emotion component, learned jointly from paired data and kept frame-synchronous, so that emotion becomes an additive, per-frame quantity that can be scaled globally or locally. However, as discussed in Related work Line 121-145, existing emotion-control methods instead act on the conditioning or internal activations (labels, prompts, speaker or emotion embeddings, steering vectors), which yields neither time-aligned emotion components nor an interpretable emotion velocity. We would like to clarify the role of each part. The velocity decomposition is the core of EmoScale: it represents emotion as a velocity component that is frame-synchronous with the semantic component and interpretable (Figure 2), which is what enables frame-level and localised intensity control. WORLD is used only to remove the comb-like artifacts that appear when the two endpoints are mixed directly; on its own, WORLD interpolation cannot cleanly separate the emotional from the neutral content.

To verify this, we replaced our endpoints with neutral and emotional versions generated independently by an existing emotional TTS model (CosyVoice3-instruct, with the real neutral recording as the speaker prompt), aligned them by DTW, and applied exactly the same pitch-aligned interpolation (Table below). Independently generated endpoints do not provide a clean emotional direction: the generated emotional end is not more emotional than the neutral end, the intensity axis is flat. They are also not time-synchronous (duration ratio 1.55), so the content breaks down at intermediate α (CER 0.26 at α = 0.5), and segmented α does not move the intensity in the intended direction. EmoScale, in contrast, yields a monotonic intensity axis and correct localised control.

MetricCosyVoice3 endpointsEmoScale (ours)
RA of the emotional end (neutral end)−0.17 (−0.06)0.69 (−0.03)
Normalised RA at α = 0 / 0.5 / 1−0.43 / −0.50 / −0.55−0.11 / 0.33 / 0.95
CER at α = 0.50.260.18
Segmented α (0 → 1): RA of 1st / 2nd half0.15 / −0.510.25 / 0.57

Endpoint-source ablation on the RAVDESS utterances of Table 3, using identical pitch-aligned interpolation.

AuthorsOpenReview ↗

Response to Reviewer kssy (2)

W3. request for more ablation study

Both DiTs and both conditioning signals are integral to the decomposition rather than optional components. Removing DiTΔ leaves a single DiT conditioned on Whisper features that can only reconstruct neutral speech, so there is no emotion component to scale, which will be largely similar to Cosyvoice component, which we added the comparison test above. Conversely, vΔ is trained as the residual between the emotional and the neutral velocity, so integrating it alone does not produce speech; it is meaningful only on top of vA. A single DiT that switches between neutral and emotional conditions can produce the two endpoints, but not a frame-wise additive emotion component, so it cannot provide frame-level α control or the interpretable emotion velocity shown in Figure 2. For the conditioning signals, sharing one adapter between the two DiTs severely degrades content preservation (Table below).

VariantCER at α = 0Overall CER
Shared conditioning (one adapter for both DiTs)1.240.410
Separate conditioning (ours)0.190.207

Regarding orthogonality, RA is Table 4 is sensitive, and the score for neutral should be as close as to the Real recording and alpha=1 should be as close as to the emotion recording. The λ = 1 provides the best performance. Additionally, adding orthogonality also provides interpretability demonstrated in Figure 2, which the current generation models have no such property. We provide both theoretical justification and empirical evidence that emotional disentanglement can be achieved, and further demonstrate its practical applicability through localised emotion-intensity control (Line 411).

In a nutshell, all the components are necessary and proved accordingly.

W4. analysis of the time-dependent velocity fields

We argue that the time-dependent velocity fields provide no further new intake against our core contribution because eventually the disentanglement has been achieved. However, as requested by reviewer, to give them a clearer overview, we report the a timestep-wise analysis of the model. We probe it in a teacher-forced manner on the full ESD test sets and report, at each timestep t, the prediction error, the relative magnitude of the two velocities and their relationship.

tvA vs. neutral target (neutral state)vA + vΔ vs. emotional target (emotional state)cos(vΔ, true emotional difference)
0.10.650.480.67
0.50.430.530.70
0.90.810.940.72

It shows that direction follows the true emotional velocity difference and the neutral and emotion velocity is learning towards the final state.

W5. Localised emotion control quantitative evaluation

We argue that the quantitative evaluation over localised emotion control has not been widely established and visualisation in Appendix F together with quantitative RA score marked in Figure 4 are enough evidence to show the effectiveness of our method. However, as requested by reviewer, we design a slicing-window experiment to provide additional quantitative results.

Setting α to 0 and 1 for the two halves moves only the target half to the intended intensity while the other half stays at its own level, in both directions. A sliding-window analysis (1 second window, 0.25 second hop;) shows that the transition completes within the analysis window: the transition progress at −0.25 / 0 / +0.25 second around the boundary is 0.29 / 0.58 / 0.80, close to 0.25 / 0.50 / 0.75 for an ideal step smoothed by the window. Perceptually, segmented utterances sound as natural as uniform ones; examples are provided on the demo page: https://anonymous.4open.science/r/emoscale-9A11/samples/index.html.

AnalysisSetting / Time relative to boundaryRA / Transition progress
Segment-level control0 → 1 (1st half)0.08
Segment-level control0 → 1 (2nd half)0.89
Segment-level control1 → 0 (1st half)0.96
Segment-level control1 → 0 (2nd half)−0.06
Sliding-window analysis−0.25 s0.29
Sliding-window analysis0 s0.58
Sliding-window analysis+0.25 s0.80
AuthorsOpenReview ↗

Response to Reviewer kssy (3)

W6. Limited training scalability

We view data efficiency as a strength of the framework rather than a limitation. EmoScale is trained on only neutral–emotional pairs from ESD, yet it generalises to three out-of-domain datasets (RAVDESS, CREMA-D and IEMOCAP) with different speakers, recording conditions and emotion distributions (Table 1). The current scale is therefore sufficient for the task, and scaling to large in-the-wild corpora is beyond the scope of this paper.

Minor weakness

We will revise Figure 1 to clarify the arrows and the relations in the inference pipeline.

Q1,3,4,5 have already been responded in previous weakness point.

Q2. direct velocity scaling

We did evaluate direct scaling in velocity space, i.e. integrating v = vA + α vΔ. As the direct interpolation has been discarded, we did not show in the main text. However, we show the results here to give a clear comparison between direct velocity interpolation with our method. The neutral and emotional endpoints have different F0 contours, so at intermediate α their harmonics are misaligned and two harmonic combs are superimposed, which is heard as comb-like artefacts. Pitch-aligned interpolation first gives both endpoints a shared F0 contour and only then mixes them, which removes this effect (the following Table).

DatasetMethodα = 0.25α = 0.5α = 0.75
ESD-zhDirect velocity scaling1.881.612.16
ESD-zhPitch-aligned (ours)2.152.072.19
ESD-enDirect velocity scaling2.682.262.90
ESD-enPitch-aligned (ours)2.832.822.93
RAVDESSDirect velocity scaling2.361.982.53
RAVDESSPitch-aligned (ours)2.392.402.57

UTMOS (↑) of direct velocity scaling vs. pitch-aligned interpolation at intermediate intensities on the full test sets

Q6. artefacts failure localisation

We claim that this localisation does not make sense because: (1) There is no commonly agreed definition of an artefact in emotional speech, and whether a perceptual characteristic counts as one is largely subjective. (2) Qualitative judgements also differ across listeners: in our own listening, alongside the real recordings and the baselines, we did not perceive these segments as artefacts.

However, as requested by reviewer, on top of ablation we performed for pure twin-dual generation process and pure WORLD without twin-dual, we ran a stage-wise analysis on the RAVDESS utterances of Table 3 and on ESD test utterances to even out the effect of waveform reconstruction. Each stage of the pipeline was applied separately, to real recordings and to our generated outputs, and we measured the change it causes (Table below):

Experimental settingsΔUTMOSΔCER
vocoder copy-synthesis of real neutral recordings−0.02 / −0.10 / −0.050.000 / −0.001 / −0.002
vocoder copy-synthesis of real emotional recordings+0.03 / −0.03 / −0.07−0.003 / +0.003 / +0.001

Waveform reconstruction is not a source. Vocoding real recordings (copy-synthesis) changes UTMOS by at most 0.10 and leaves CER unchanged. Essentially we still claim there are no artefacts.

W7. novelty (Again)

Besides aforementioned novelty, we summarise there again (though we have already stated in Line99-161):

What is decomposed. Residual-based emotion control typically adds an emotion offset to the conditioning, and disentangled flow-matching approaches separate factors in the conditioning or latent space; in both cases the generative velocity remains a single, entangled field. EmoScale instead decomposes the velocity field itself. The emotion component is therefore defined directly in the space in which speech is generated.

Twin-forward mechanism. Both components are predicted on the same noise and time grid, so the emotion contribution is frame-synchronous with the content and additive. This allows frame-level and localised intensity control and makes the emotion velocity directly observable: vΔ follows the true emotional velocity difference (cosine 0.67–0.89) and clusters by emotion rather than by speaker or sentence (Figure 2).

Role of WORLD. WORLD aligns the pitch of the two synchronous endpoints before mixing to avoid comb artefacts. When the same interpolation is applied to endpoints generated by an existing emotional generation model, the intensity axis collapses.

In conclusion, we would like to thank the reviewer again. We believe that most of the concerns stem from potential misunderstandings, rather than fundamental issues with the research. We kindly encourage the reviewer to reconsider these points and the overall score.

Reviewer_6VpnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EmoScale for continuous control of emotional intensity in speech, including localised adjustment within user-specified temporal regions of an utterance. The core idea is to learn a decomposition of the velocity field of a DiT-based flow-matching model into a semantic component for neutral speech generation and an emotion-residual component.

Critical strengths

S1. I particularly value the application of a disentanglement constraint directly to the generator’s output space, namely the predicted velocity fields, rather than relying solely on conditioning embeddings for emotion disentanglement.

S2. The paper evaluates the effectiveness of emotion-intensity control across multiple corpora and through subjective evaluation. Including three out-of-domain corpora and human judgements, rather than relying only on automatic scores from a single dataset, is a strength.

Critical weaknesses

W1. The evidence for content preservation and the effectiveness of orthogonality regularisation is insufficient. In Table 3, EmoScale has higher CER than the baseline models under the weak and medium intensity conditions. This trade-off between emotion-intensity control and content correctness raises an important concern regarding the central claim of preserving linguistic content, yet the sources of these errors are not sufficiently analysed. Furthermore, the ablation study in Table 4 is limited to geometric separation metrics and RA scores. It therefore remains unclear what practical benefits orthogonality regularisation provides for content preservation or speech quality when emotional intensity is varied.

W2. Figures 2 and 3 provide qualitative evidence for interpretability, but the details of representation extraction, dimensionality reduction, standardisation, and aggregation, as well as the criteria used to assess controllability, are not sufficiently explained. In addition, presentation issues, such as the duplicated wording “Adapter adapter” and the inconsistent use of "x_e" and "x_emo", reduce readability. The provided audio demos do not appear to include corresponding samples from baseline models under comparable conditions, making direct perceptual comparisons difficult. Furthermore, the generated samples exhibit noticeable degradation in speech quality and linguistic intelligibility.

W3. Direct comparisons with existing steering approaches are limited. Although Table 3 includes EmoKnob, which controls emotion intensity by manipulating speaker embeddings, it does not include direct comparisons with the activation- or hidden-state-based methods EmoSteer-TTS [1] and DUET [2] discussed in Related Work. Based on the current results, it is difficult to determine what additional benefits the proposed training-time velocity decomposition provides over these related steering approaches in terms of emotion-intensity control and content preservation.

[1] Xie, Tianxin, et al. "Emosteer-tts: Fine-grained and training-free emotion-controllable text-to-speech via activation steering." arXiv preprint arXiv:2508.03543 (2025). [2] Zhang, Xu, Longbing Cao, and Zhangkai Wu. "DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech." arXiv preprint arXiv:2606.00066 (2026).

Questions for authors

Q1. Given that emotional expression involves multiple acoustic attributes, is the proposed pitch-aligned intensity interpolation sufficient for emotion-intensity control? Although it adjusts not only F0 but also the spectral envelope and aperiodicity, it remains unclear how well jointly varying these attributes within a fixed temporal structure captures intensity changes across different emotions.

Q2. The description of the subjective evaluation specifies the numbers of participants and utterances, but could the authors clarify how the BWS comparison sets were constructed, whether each set compared intensity variants with the same sentence content, speaker, and emotion, and how the scores were aggregated? Table 2 supports the overall ordering of emotional intensity, but the BWS differences between the intermediate levels alpha=0.5 and alpha=1.0 are relatively small across all datasets.

10 comments
AuthorsOpenReview ↗

Response to Reviewer 6Vpn

Thanks to you the reviewer to spend efforts to provide the suggestions. We have respond below point by point to clarify potential misunderstandings.

W1. Content preservation quality and Orthogonality gains

(i) CER. ASR-based CER is only a proxy for content preservation, and in our outputs the errors are not spread uniformly: they are concentrated in a small subset of speakers and utterances. CER is commonly regarded as a coarse measure of content preservation once it is already low (around below 0.5 in most of our experimental settings) as acknowledged by previous works (https://arxiv.org/abs/2510.26190). We argue this may reflect a limitation of current automatic quality metrics rather than a drop in generation quality, and therefore view this as a broader limitation in assessing expressive speech generation, and we treat UTMOS and CER as auxiliary checks, supplemented by human evaluation in Table 2. The content remains intelligible on listening, and we kindly invite the reviewer to listen to the samples, now including the baselines, on the anonymous demo page: https://anonymous.4open.science/r/emoscale-9A11/samples/index.html.

(ii) Orthogonality. The orthogonality regulariser is the expression of our core design while training. EmoScale decomposes the velocity field into a semantic component vA, which carries the linguistic content, and an emotion component vΔ, which carries the emotional deviation from it. The two describe different factors and are therefore constrained to be orthogonal. Beyond the interpretability demonstrated in Figure 2 and the ablation study in Table 4, which directly validate our methodological contributions, our work offers broader insights to the research community. Practically specifically, we provide both theoretical justification and empirical evidence that emotional disentanglement can be achieved, and further demonstrate its practical applicability through localised emotion-intensity control (Line 411).

W2. visualisation experimental details, typos, and quality concerns

We summarise the details of Figures 2 and 3 in the following Table and will add them to the appendix. The typos such as duplicated “Adapter adapter” and the inconsistent use of xe / xemo will be corrected in the revision. We have also added samples of the Table 3 baselines EmoKnob and CosyVoice3 (IC and CC), generated under the same conditions, to the anonymous demo page, so that changes in emotion and content can be compared directly.

FigureRepresentationStandardisation and aggregationDimensionality reduction and criterion
2(a)Emotion velocity vΔ predicted at the midpoint of the flow (t = 0.5), averaged over time frames: one 80-dim vector per utterance and αNone (raw vectors)UMAP; we show the 2-D view with the clearest emotion structure. Silhouette score by emotion +0.30 vs. by speaker +0.05
2(b)Trajectory points along the integration steps, averaged over time frames (one 80-dim vector per step)NonePCA, first two principal components
388-dim eGeMAPSv02 functionals (openSMILE) per utterancez-scored with the normalisation statistics of the RA ranker; median over utterances at each αA feature is controllable for an emotion if its median curve over the six α values (0–1.25) moves in the same direction in at least 4 of the 5 steps; features are ranked by the number of emotions (0–4) for which this holds

Regarding content degradation, together with what we claim before about the limitation of CER, we claim UTMOS related quality automatic metrics have limitation as well, so its absolute values should be interpreted with care. Emotion itself lowers UTMOS. On real recordings (Table 1), UTMOS drops from neutral to emotional speech: ESD-zh 3.68 → 2.74, ESD-en 4.19 → 3.65, RAVDESS 4.08 → 3.32.

UTMOS under-scores real speech. UTMOS is calibrated on English synthetic speech; a recent study reports median scores of only 2.72 and 2.43 on clean read Greek and English recordings, respectively [arXiv:2609.13498].

Since EmoScale outputs emotional speech by design, part of the gap reflects this metric bias rather than audible degradation. In our listening, outputs above a moderate UTMOS level are perceptually on par with the baselines. Many previous speech generation works have also stated such metrics are not very reliable such as https://arxiv.org/pdf/2203.11389. We stated here again that this may reflect a limitation of current automatic quality metrics rather than a drop in generation quality, and therefore kindly refer the reviewer to hear the samples at https://anonymous.4open.science/r/emoscale-9A11/samples/index.html.

AuthorsOpenReview ↗

Response to Reviewer 6Vpn (2)

W3. more baseline model comparison

Neither EmoSteer-TTS nor DUET provides official code, so we initially did not include them as the current results in Table 3 are sufficient evidence of effectiveness of our models. However, to give the reviewer a clearer overview, we find out that both are training-free and operate on the public F5-TTS backbone, so we reimplemented both on F5-TTS following the configurations reported in their papers. For EmoSteer-TTS we use the steering layers, steps and token selection given in its Appendix C; for DUET we steer layer 16 as reported and calibrate the unreported hyper-parameters ourselves. Steering directions are estimated on the ESD training split only. We evaluate both methods on the same RAVDESS utterances under the same protocol of Table 3, using the real neutral recording as the speaker prompt and three intensity levels at 0.5 / 0.75 / 1.0 of each method’s full strength, and report RA, CER, and UTMOS in the following Table.

MethodRA (W / M / S)CER (W / M / S)UTMOS (W / M / S)
EmoSteer-TTS (reimplemented)0.43 / 0.53 / 0.660.020 / 0.024 / 0.0263.54 / 3.10 / 3.07
DUET (reimplemented)2.54 / 3.16 / 2.860.034 / 0.196 / 0.7142.14 / 1.52 / 1.37
EmoScale (ours)0.40 / 0.58 / 0.790.180 / 0.094 / 0.0642.40 / 2.57 / 2.61

Activation-steering baselines (our reimplementations on F5-TTS) vs. EmoScale on the RAVDESS utterances of Table 3. W / M / S: weak / medium / strong. For reference, F5-TTS without steering obtains RA 0.28.

It shows consistent results in our paper. EmoSteer has narrowed intensity range and DUET levels are less consistently ordered. The comparison shows that velocity decomposition provides a wider and more consistently ordered intensity axis.

Q1. Concern on attributes control

Our answer for whether such interpolation over attribution is sufficient is Yes. WORLD decomposes speech into F0, spectral envelope and aperiodicity, from which the waveform is fully reconstructed, showing that these attributes are sufficient enough for generating a new audio. Moreover, varying them with α also changes other acoustic features derived from it, as we have proved in Figure 3 and Appendix D. In particular, the pitch-, loudness- and spectral-envelope-related eGeMAPS descriptors, also change monotonically with α across the investigated emotions. Therefore, we are not only revising F0, or just tiny few features.

Q2. BWS settings

Each BWS comparison set contained four versions of the same utterance, with the same sentence, speaker and emotion, generated with α ∈ {0, 0.5, 1.0, 1.5}; participants selected the version with the strongest and the version with the weakest emotional intensity. For each dataset we used 10 such sets (40 utterances), i.e. 50 sets over the five datasets; each participant judged 15 sets (60 utterances), giving three judgements per set on average. The score of each item is (#best − #worst) / #appearances, averaged per α level, and significance was assessed with a one-way ANOVA across α levels. Therefore, the best and worst samples will likely be chosen more times, leaving the small gap between α = 0.5 and α = 1.0 inherently natural. And in other words, in a four-item set, the two middle items are rarely chosen as the most or the least intense, so their scores naturally concentrate around zero. BWS mainly separates the extremes, while the ordering of the two middle levels (0.5 < 1.0) is still preserved on all five datasets (Table 2).

In conclusion, we would like to thank the reviewer again for raising these concerns and giving us the opportunity to clarify our work. We believe that most of the concerns stem from potential misunderstandings, rather than fundamental issues with the research. We kindly encourage the reviewer to reconsider these points in light of our responses, and kindly ask the reviewer to reconsider the overall score in light of these clarifications.

Reviewer_6VpnOpenReview ↗

Official Comment by Reviewer 6Vpn

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns. However, I would like to clarify a few remaining points.

1. Content Preservation and Orthogonality Regularisation While I agree that UTMOS has limitations, I am not fully convinced that the limitations of CER justify the reported results. In particular, considering CER below 0.5 as low differs substantially from recent TTS benchmarks, where models typically achieve CER values around 0.01, as reported in the FireRedAudio evaluation (https://github.com/FireRedTeam/FireRedAudio/blob/main/README.md?utm_source=chatgpt.com). Moreover, Table 3 shows that EmoScale has a CER of 18.0% under weak intensity, compared with 1.8% for CosyVoice3, although their achieved emotion intensities differ. If the authors attribute these results to limitations of the evaluation metrics, additional evidence would be helpful, including an error analysis, content preservation evaluations across different λ values, and complementary speech quality metrics such as UTMOSv2 or DNSMOS where appropriate.

2. Audio Demo Comparisons I appreciate the authors' efforts to include additional comparisons with EmoSteer-TTS and DUET. Could the authors also provide corresponding audio samples from EmoSteer-TTS, DUET, and their underlying F5-TTS backbone on the demo page? Additionally, providing loudness-matched versions alongside the original samples would help facilitate fairer perceptual comparisons of speech quality without removing loudness differences that may contribute to emotional expression.

Reviewer_RaGdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for continuous emotion-intensity control. This paper utilizes twin DiT decoders, one for 'semantic' modeling and another for 'emotion' delta modeling. Orthogonal regularisation is applied during training with the paired ESD dataset. The experimental results show that the proposed model could control the emotion of speech by linearly interpolating on extracted speech features like F0.

Critical strengths

Linearly interpolating for DiT-based model is interesting.

Critical weaknesses
  1. The task is emotional speech re-synthesis or editing, not speech generation, and the baseline comparison is mismatched.
  2. Speech quality and content preservation are weak. As in Table 3, the UTMOS score is about 2.0- 3.0, compared with 3.7- 4.2 for the real recordings or other models.
Questions for authors
  1. Can EmoScale generate emotional speech from text alone, or does it always require an emotional recording of the same utterance as input?
  2. What happens if intensity is controlled directly in velocity space rather than by WORLD morphing?
5 comments
AuthorsOpenReview ↗

Response to Reviewer RaGd

Thanks to you the reviewer to spend efforts to provide the suggestions. We have respond below point by point to clarify potential misunderstandings.

W1: Definition of tasks and baseline choice.

We agree that EmoScale is an emotion-intensity editing method, which is also a form of generation (https://arxiv.org/abs/2403.16973 has discussed speech editing and generation in parallel and https://arxiv.org/abs/2310.00704 a generation LLM has listed speech edition as one of the important tasks. Specifically, we define our tasks as emotion-intensity editing method as it takes an emotional recording as input and produces a continuum from a neutral rendition (α = 0) to the original emotion (α = 1) and beyond.

We selected relevant speech generation baseline because 1. our core claim is emotion intensity control which is widely discussed in generation models, and all the baselines we select make emotion control as central contribution claim in Table 3; 2. to our knowledge few of them expose an explicit intensity control, so a comparison would not test our core claim and would offer limited insight.

However, following the suggestion, we additionally evaluated AudioEditor (Jia et al., 2024), a training-free diffusion-based audio editor, on the same RAVDESS utterances, using its official “add” operation with the prompts “with slight / moderate / strong {emotion}”. As a general-purpose audio editor, it does not preserve the linguistic content of speech (Table R1); the content is lost in the editing step itself, since inversion-only reconstruction keeps it intact. We therefore report AudioEditor as a general editing reference rather than an intensity-control baseline. We also attempted Emovox (Zhou et al., 2022), an intensity-controllable emotional voice conversion model, but its pre-trained weights are not available (the released Stage-I link is broken and the released checkpoint is incomplete), so it could not be reproduced as we cannot access its training setups and dataset.

ModelSettingRA ↑UTMOS ↑
AudioEditorInversion only (no edit)0.011.74
AudioEditorWeak0.151.32
AudioEditorMedium0.181.46
AudioEditorStrong0.401.35
EmoScale (Ours)Weak0.402.40
EmoScale (Ours)Medium0.582.57
EmoScale (Ours)Strong0.792.61

It shows that from either generation quality and emotion intensity control, our EmoScale performed better.

W2. Speech quality and preservation:

We respectfully note that UTMOS systematically under-scores emotional speech, so its absolute values should be interpreted with care. Emotion itself lowers UTMOS. On real recordings (Table 1), UTMOS drops from neutral to emotional speech: ESD-zh 3.68 → 2.74, ESD-en 4.19 → 3.65, RAVDESS 4.08 → 3.32.

UTMOS under-scores real speech. UTMOS is calibrated on English synthetic speech; a recent study reports median scores of only 2.72 and 2.43 on clean read Greek and English recordings, respectively [arXiv:2609.13498].

Since EmoScale outputs emotional speech by design, part of the gap reflects this metric bias rather than audible degradation. In our listening, outputs above a moderate UTMOS level are perceptually on par with the baselines. We have added baseline samples generated under identical conditions to the anonymous demo page and kindly invite the reviewer to compare them directly. Many previous speech generation works have also stated such metrics are not very reliable such as https://arxiv.org/pdf/2203.11389. We argue this may reflect a limitation of current automatic quality metrics rather than a drop in generation quality, and therefore view this as a broader limitation in assessing expressive speech generation, and we treat UTMOS and CER as auxiliary checks, supplemented by human evaluation in Table 2. The samples are available at https://anonymous.4open.science/r/emoscale-9A11/samples/index.html.

Q1. Clarification of EmoScale input setting

No. EmoScale requires the emotional recording of the utterance; it needs neither the transcript nor a neutral recording at inference time. The neutral rendition (α = 0) and all intermediate intensities are generated from the emotional input alone.

AuthorsOpenReview ↗

Response to Reviewer RaGd (2)

Q2. Direct interpolation with velocity

We did evaluate direct scaling in velocity space, i.e. integrating v = vA + α vΔ. As the direct interpolation has been discarded, we did not show in the main text. However, we show the results here to give a clear comparison between direct velocity interpolation with our method. The neutral and emotional endpoints have different F0 contours, so at intermediate α their harmonics are misaligned and two harmonic combs are superimposed, which is heard as comb-like artifacts. Pitch-aligned interpolation first gives both endpoints a shared F0 contour and only then mixes them, which removes this effect (the following Table).

DatasetMethodα = 0.25α = 0.5α = 0.75
ESD-zhDirect velocity scaling1.881.612.16
ESD-zhPitch-aligned (ours)2.152.072.19
ESD-enDirect velocity scaling2.682.262.90
ESD-enPitch-aligned (ours)2.832.822.93
RAVDESSDirect velocity scaling2.361.982.53
RAVDESSPitch-aligned (ours)2.392.402.57

UTMOS (↑) of direct velocity scaling vs. pitch-aligned interpolation at intermediate intensities on the full test sets

In conclusion, we would like to thank the reviewer again for raising these concerns and giving us the opportunity to clarify our work. We believe that most of the concerns stem from potential misunderstandings, rather than fundamental issues with the research. We kindly encourage the reviewer to reconsider these points in light of our responses, and kindly ask the reviewer to reconsider the overall score in light of these clarifications.

AuthorsOpenReview ↗

Reminder to join rebuttal discussion

Dear Reviewer RaGd, We kindly invite you to review our rebuttal since the discussion phase is ending soon. We have carefully addressed all your concerns, including baseline comparison and quality metric limitation.

We have discussed with another reviewer 6Vpn and have agreed that the CER and UTMOS are flawed, and provided additional metrics DNSMOS as proof for objective metric evidence. We hope that could add upon our explanation regarding your second weakness point concern.

If you have any further questions or would like additional clarifications, we would be happy to continue the discussion.

Thanks a lot and we look forward to the discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.