acceptodds
Under review as a conference paper at ICLR 2027

EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation

Abstract

Emotion-conditioned text-to-speech (TTS) models may fail to express the requested emotion reliably, and improving controllability by additional training is costly in both computation and emotion-labeled speech training data. We therefore study vector steering, a training-free approach that modifies the internal representations of a frozen model. CoCoEmo, a conventional vector steering method for emotion TTS, treats each emotion vector as an indivisible direction controlled by a single global strength, limiting adherence to the requested emotion. In this work, we first discover that an emotion vector can be decomposed into a shared component that moves speech away from neutral expression and a residual component that directs generation toward the requested emotion. Building on this finding, we propose Emotion Residual-Enhanced Steering (EmoRES), a novel method that controls the two components without retraining the backbone. On IEMOCAP, EmoRES outperforms CoCoEmo across all four objective emotion metrics on the IndexTTS-2 and CosyVoice2 backbones. Rank correlation improves by 26.13 and 12.97 percentage points, corresponding to relative gains of 118.8% and 33.1%, while emotion hit rate improves by 12.95 and 6.92 points, corresponding to relative gains of 20.1% and 9.8%. Human evaluation further shows a relative improvement up to 35.0% in the rate at which listeners correctly identified the dominant requested emotion and up to a 17.3% improvement in fidelity, while listeners prefer EmoRES for naturalness in up to 63.8% of pairwise comparisons. Component ablations further demonstrate that effective control benefits from preserving the shared component while strengthening the residual of the emotion steering vectors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech, Text-to-speech, Activation steering, Emotion, LLM

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_jjBUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free steering method to improve emotion control in TTS. The key idea is to decompose a conventional emotion steering vector into a shared component which represents general emotionality and a residual component which encodes specific emotion information. Therefore, the two vectors could be weighted independently for more precise control without fine-tuning the TTS backbone. Experiments on IndexTTS-2 and CosyVoice2 show that the proposed method achieves more accurate emotion control while preserving speech naturalness and speaker similarity.

Critical strengths
  1. The method is simple but practically useful. The proposed method first separates the emotion vector into shared and specific residual components, and then controls them independently. This formulation is easy to understand and requires no retraining of the TTS backbone.
  2. The experimental analysis is well aligned with the main claim. Results on two different TTS backbones, together with OOD and human evaluations, make the empirical evidence reasonably convincing.
Critical weaknesses
  1. The methodological novelty is somewhat limited. The vector extraction procedure and overall inference framework are largely inherited from CoCoEmo. While the resulting improvement is meaningful, the method advance itself is fairly incremental.
  2. The functional interpretation is not fully validated. The split is defined using an unweighted centroid over only a few emotion categories, so it is unclear whether the same interpretation holds with more emotions or model families.
Questions for authors
  1. How sensitive is the decomposition to the choice of emotion categories used to compute the centroid? For example, would the conclusions remain similar if the emotion set were changed or expanded?
  2. The main experiments use λc=1\lambda_c=1 and λr=3\lambda_r=3. How robust is this setting across different emotions and backbones?
1 comment

Weakness 1: Methodological novelty relative to CoCoEmo

We thank the reviewer for recognizing that the improvement is meaningful and practically useful.
Reusing CoCoEmo’s pipeline for fair comparison. We adopted CoCoEmo’s extraction, layers, and injection rule deliberately, so the comparison is controlled: every difference comes from how the steering vector is composed. EmoRES itself is not tied to this pipeline. Applied to EmoSteer-TTS, which steers the flow-matching module instead of the speech LM, it improves all four emotion metrics on CosyVoice2 (results in Table 4 of revised paper).

Contribution. CoCoEmo establishes where to steer. We examine what the emotion vector contains. Each vector splits into a shared component that moves speech away from neutral and a residual that specifies the emotion. Conventional composition of CoCoEmo fixes the ratio between the two. Our EmoRES method makes it controllable.
CoCoEmo cannot match EmoRES. Increasing CoCoEmo’s steering strength α\alpha saturates the emotion metrics beyond α=3\alpha = 3 while S-SIM and WER degrade (Fig. 1). With the injected norm held equal, EmoRES wins at every α\alpha with IndexTTS-2. On held-out IEMOCAP, ρ improves from 22.00% to 48.13%, and human Dom-hit improves by about 19 points.
Generality. Beyond CoCoEmo’s two backbones, performance gains also hold on CosyVoice3 and Qwen3-TTS (Appendix E) and across three SER evaluators (Appendix H).


Weakness 2 and Question 1: Sensitivity of the decomposition to the emotion set

In the experiments of the original manuscript, we used 4 emotions to compute the centroid. To test the sensitivity of the choice of emotion categories to compute the centroid of emotion vectors, besides the original four emotions (angry, happy, sad, surprised), we added two additional emotions, fear and disgust. Table H shows the performance difference between the set of emotions used to construct the mean vector. On the CREMA-D evaluation set (angry, happy, sad), for EmoRES, the mean values of E-SIM, TEP, ρ, and H-rate drop a little when using more emotions for vector construction. When comparing EmoRES with CoCoEmo, no matter using 4 or 6 emotions, EmoRES outperforms CoCoEmo on ρ and H-rate by a large margin, demonstrating that the conclusion remains similar if the emotion set were expanded.
Table J shows the performance of EmoRES using the 6-emotion centroid on the RAVDESS dataset (6 emotions). EmoRES also outperforms CoCoEmo on E-SIM and TEP by a large margin.

Table H. Ablation on expanding the emotion set for computing the centroid from 4 emotions to 6 emotions for EmoRES with the CosyVoice 2 backbone on the CREMA-D evaluation set. For CoCoEmo, since the steering formula of Eq. (2) is independent of the emotion centroid, the results of the original emotions do not change when new emotions are used to calculate the emotion centroid.

MethodCentroidE-SIM↑TEP↑ρ↑H-Rate↑
No steer-59.70±0.3634.01±0.96--
CoCoEmo (λc=1,λr=1λ_c=1, λ_r=1)-74.16 ± 0.0967.43 ± 0.4032.48 ± 2.1778.92 ± 0.54
EmoRES (λc=1,λr=3λ_c=1, λ_r=3)4 emotion74.71 ± 0.2468.80 ± 0.5651.94 ± 6.3084.76 ± 2.07
EmoRES (λc=1,λr=3λ_c=1, λ_r=3)6 emotion74.12 ± 0.4567.59 ± 0.9051.23 ± 4.3983.89 ± 1.85

Table J. Expanding the emotion set for computing the centroid with 6 emotions for EmoRES with the CosyVoice 2 backbone on the RAVDESS evaluation set (6 emotions). Since RAVDESS only has one emotion per clip, ρ and H-rate do not apply to this setting.

MethodE-SIM↑TEP↑
Unsteered57.67 ± 0.1225.19 ± 0.31
CoCoEmo (λc=1,λr=1λ_c=1, λ_r=1)76.62 ± 0.1254.78 ± 0.85
EmoRES (λc=1,λr=3λ_c=1, λ_r=3)80.20 ± 0.5060.64 ± 0.41

Question 2: Robustness of the coefficient setting across emotions and backbones

Robustness across different emotions: EmoRES achieves higher mean ρ, H-Rate, TEP and E-SIM than CoCoEmo for angry-, happy- and sad-dominant IEMOCAP requests on IndexTTS-2. These consistent gains support the applicability of a common coefficient setting across the tested emotion groups, without emotion-specific tuning.

Table K. Per-emotion improvements of EmoRES compared to CoCoEmo on IEMOCAP with IndexTTS2 under the setting of λc=1λ_c=1 and λr=3λ_r=3.

EmotionΔρ ↑ΔH-Rate ↑ΔTEP ↑ΔE-SIM ↑
Angry+27.27 ± 5.00+13.09 ± 2.63+4.84 ± 0.90+3.57 ± 0.72
Happy+12.45 ± 8.50+5.45 ± 3.37+1.87 ± 0.95+0.75 ± 0.18
Sad+35.01 ± 5.88+17.45 ± 2.65+8.90 ± 0.93+2.25 ± 0.29

Robustness across different backbones: λr=3λ_r=3 was selected only on CREMA-D and applied unchanged everywhere else. It improves multiple emotion-related metrics over CoCoEmo on 4 backbones with different conditioning interfaces (Tables 1,6,7), on held-out IEMOCAP, with emotional speaker prompts (Table 8), without native emotion conditioning (Table 9), and under three independent SER evaluators (Table 10).

Reviewer_NBRiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a training-free feature manipulation method for emotion-controllable TTS. Starting from extracting emotion vectors from paired emotion-neutral feature differences, the method decomposes each vector into a shared component, defined by the average across emotion vectors, and an emotion-specific residual. It independently reweights these components, emphasizing the residual while retaining the shared component. Experiments with frozen TTS backbones, ablation studies, qualitative results, and human judgements demonstrate improved emotional control over conventional steering.

Critical strengths
  • The method is simple and practical. EmoRES reuses an existing vector feature without updating backbone parameters. The method is straightforward to implement.
  • Component ablation study is plausible. Norm-matched comparisons help distinguish directional improvements from increased perturbation magnitude. Component removal and randomization provide evidence that the measured directions matter. Experiments without native emotion conditioning further support an effect attributable to steering itself.
  • The method shows generalization ability. The method shows the improvement with cross-dataset evaluation settings and multiple backbones.
Critical weaknesses

W1. The semantic interpretation is stronger than what the decomposition establishes.

  • Decomposing a vector into its across-emotion mean and a centered residual is an algebraic identity; it does not itself establish a separation between general emotionality and emotion identity. The ablations support different functional contributions, but reducing the neutral posterior does not rule out a bias toward particular non-neutral emotions. Likewise, a component being identical across requests does not imply that its effect is category-neutral.

W2 (minor). The adequacy of a single fixed direction per emotion is insufficiently characterized.

The paired construction controls for speaker and lexical identity, making the mean difference a reasonable estimator. However, emotion-dependent interaction with speaker, content, and expression style may remain. The paper demonstrates the resulting vectors are useful, but provides limited anlysis of their stability across data subsets or how well a single vector represents the distribution of paried activation differences. This matters particularly for the proposed interpretation of the components, although the extraction procedure is inherited from CoCoEmo.

W3. The methodological contribution is relatively incremental.

Its empirical benefits are valuable, but the stronger scientific contribution depends on how convincingly the proposed functional interpretation is established.

Questions for authors
  • Under shared-only steering, how does the complete emotion posterior distribution changes? Does the shared component consistently promote non-neutral emotions, or does it dispropotionately favor particular categories? Would its interpretation remain stable if the emotion categories used to compute the centroid changes?

  • How stable are the emotion vectors when estimated independently from different speakers or corpora?

  • (Minor) Does residual enhancement remain beneficial when the underlying emotion directions are obtained using alternative estimators, such as robust means or suitably constructed SVD based directions? Such comparisons could clarify whether the proposed interpretation generalized beyond the particular mean-difference vector library.

2 comments
AuthorsOpenReview ↗

Part (1/2)

Weakness 1 and Question 1: Functional interpretation of the shared component

Table M. Non-neutral prediction rate and mean emotion posteriors (%) from emotion2vec+ large for unsteered and shared-only CosyVoice2 generation on CREMA-D.

MeasureUnsteeredShared-onlyΔ (pp)
Non-neutral prediction rate52.34 ± 1.6985.45 ± 1.36+33.11 ± 2.16
Posterior: Angry6.54 ± 0.618.04 ± 0.98+1.50 ± 1.22
Posterior: Sad32.81 ± 0.9143.08 ± 1.97+10.27 ± 1.89
Posterior: Happy10.70 ± 0.9722.93 ± 1.17+12.23 ± 1.95
  • How the full posterior changes under shared-only steering. In Table M, Shared-only steering raises the rate of non-neutral emotion predictions from 52.34% to 85.45% (+33.11 pp) and increases the posterior of every non-neutral emotion.
  • Consistent promotion or category bias? The shared component consistently promotes non-neutral emotion, but not uniformly: happy (+12.23 pp) and sad (+10.27 pp) rise far more than angry (+1.50 pp). It is therefore not category-neutral. What its construction guarantees is that it is request-independent: the same shared vector is applied whatever mixture is requested. This explains why shared-only steering cannot follow the requested proportions (low ρ, Table 3 in the original manuscript). Strengthening the request-dependent residual gives the request more influence relative to this fixed shift of the shared component, which is consistent with the improved mixture control. However, the shared component is still useful, since removing it entirely lowers TEP and E-SIM on both backbones (Table 3), so the best setting down-weights it without eliminating it. We will describe the shared component as a request-independent shift away from neutral, rather than as category-neutral emotionality.
  • Stability under a different set of centroid emotions. We recomputed the centroid after adding fear and disgust to the original four emotions (angry, happy, sad, surprise) and repeated the CosyVoice2 evaluation on CREMA-D with all other settings unchanged (Table H). The gains in mixture control are stable: with the 6-emotion centroid, EmoRES reaches ρ = 51.23 and H-Rate = 83.89, against 51.94 and 84.76 with the 4-emotion centroid and 32.48 and 78.92 for CoCoEmo. The main conclusion, that EmoRES improves mixture control (ρ and H-Rate) over CoCoEmo, is not sensitive to expanding the centroid from four to six emotions. For more discussions of this part, please refer to the response to Reviewer jjBU's Question 1.

Table H. Ablation on expanding the emotion set for computing the centroid from 4 emotions to 6 emotions for EmoRES with the CosyVoice2 backbone on the CREMA-D evaluation set. For CoCoEmo, since the steering formula of Equation (2) is independent of the emotion centroid, the results of the original emotions do not change when new emotions are used to calculate the emotion centroid.

MethodCentroidE-SIM ↑TEP ↑ρ ↑H-Rate ↑
No steer-59.70±0.3634.01±0.96--
CoCoEmo (λc=1,λr=1\lambda_c=1, \lambda_r=1)-74.16±0.0967.43±0.4032.48±2.1778.92±0.54
EmoRES (λc=1,λr=3\lambda_c=1, \lambda_r=3)4 emotion74.71±0.2468.80±0.5651.94±6.3084.76±2.07
EmoRES (λc=1,λr=3\lambda_c=1, \lambda_r=3)6 emotion74.12±0.4567.59±0.9051.23 ± 4.3983.89 ± 1.85

Weakness 2: Adequacy of a single direction per emotion

Stability across speakers:
According to Table L, across angry, happy, and sad, independently estimated directions from disjoint speaker subsets exhibit high cosine agreement, supporting the stability of the average emotion directions across speakers in CREMA-D.

Table L. Speaker-subset stability of CosyVoice2 emotion directions on CREMA-D. For each emotion, each cell reports Layer 14 / 17, with median cosine similarities across 1,000 random splits of speakers into two disjoint groups.

EmotionSpeakersShared component ↑Residual component ↑
Angry860.934 / 0.9250.959 / 0.957
Happy860.934 / 0.9250.869 / 0.873
Sad840.934 / 0.9250.966 / 0.966
AuthorsOpenReview ↗

Part (2/2)

Weakness 3: Contribution and functional interpretation

We agree that the scientific contribution rests on how well the functional interpretation is established, and we have added evidence for it beyond the original ablations. In summary:

Activation-space probing (new). According to results in Table A (response to reviewer km3s Weakness 1), on held-out CosyVoice2 activations, projection onto the shared direction separates neutral from emotional speech better than projection onto the residuals (AUROC 0.883 vs. 0.675 at layer 14), while the residuals classify the emotion category better (balanced accuracy 78.17% vs. 58.16%; chance 33.3%). Layer 17 shows the same pattern.

Behaviour under steering. Stated in Section 5.2 of the original manuscript, shared-only steering lowers the neutral posterior (36.51% → 7.49% on IndexTTS-2) but does not follow the requested mixture (ρ = 5.30 and 7.20 on the two backbones, Table 3). Residual-only steering, by contrast, follows the requested mixture (ρ = 45.16 and 39.94 on IndexTTS-2 and CosyVoice2), although it reaches lower TEP and E-SIM than the combined vector.

Direction-specific controls. Replacing the residual with a random direction of the same norm collapses ρ (25.24 → 5.62 on IndexTTS-2; 32.48 → 0.78 on CosyVoice2), and randomizing the shared component lowers E-SIM and TEP (Table 5), so the effects come from each component’s direction and not its magnitude.

Together with the analysis in our response to Weakness 1 and Question 1 above, this supports the revised claim: the shared component is a request-independent shift away from neutral, and the residual carries the requested emotion identity. The full probing table is in our response to Weakness 1 of Reviewer km3s.


Question 2: Stability of emotion vectors across speakers and corpora

  • For CosyVoice2, we measured:
    - Across speakers: independently estimate vectors from two disjoint speaker groups, repeat over 1,000 random partitions, and report median cosine similarity.
    - Across corpora: independently estimate each emotion vector from CREMA-D and RAVDESS, then compare their directions.

    Table N. Stability of CosyVoice2 emotion vectors across speaker subsets and corpora. Values in each cell denote the cosine similarity for layer 14 and layer 17.

EmotionCREMA-D speaker-half medianRAVDESS speaker-half medianCREMA-D ↔ RAVDESS cosine
Angry0.962 / 0.9620.898 / 0.8680.394 / 0.493
Happy0.934 / 0.9270.825 / 0.8000.455 / 0.534
Sad0.906 / 0.8910.637 / 0.6200.259 / 0.372

Results in Table N indicate emotion vectors estimated from separate speaker groups generally point in similar directions, particularly on CREMA-D. Agreement is weaker for some RAVDESS emotions, especially sadness. Vectors estimated from CREMA-D and RAVDESS are less similar, indicating that the estimated directions also depend on the dataset.
These results support using repeatable average emotion directions while recognizing dataset-specific variation. EmoRES does not require identical directions across datasets. It reweights the shared and residual components of a given vector library. Separately, the IEMOCAP results in Table 1 show improved emotion control on an unseen evaluation corpus using the original pooled library.


Question 3: Robustness to the direction estimator

To test robustness to the direction estimator, we additionally evaluated a 10% trimmed mean, which removes the most extreme values before calculating the average. Results are reported in Table G. Specifically, for each emotion and layer, we sort the values of each vector entry across paired activation differences, discard the lowest and highest 10%, and average the remaining approximately 80%. This reduces the influence of extreme activation differences on the estimated emotion direction. The shared/residual decomposition is then applied unchanged. CoCoEmo and EmoRES use the same trimmed direction library, allowing us to isolate the effect of residual enhancement.

Table G. CosyVoice2 results for the RAVDESS-derived 10% trimmed-mean direction library, evaluated on CREMA-D.

MethodE-SIM ↑TEP ↑ρ ↑H-Rate ↑
CoCoEmo64.94 ± 0.4147.11 ± 1.3915.04 ± 10.3472.07 ± 3.30
EmoRES66.35 ± 0.2050.08 ± 1.3025.10 ± 5.0375.32 ± 1.65

With RAVDESS-derived trimmed-mean directions, EmoRES improves all four mean emotion-control metrics over CoCoEmo under matched settings. This indicates that residual enhancement can remain beneficial when emotion directions are estimated using a robust mean.

Reviewer_nBEnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies training-free activation steering for emotion control in LM-based TTS. Building on CoCoEmo, which injects each mean-difference emotion vector as a single direction with one global strength, the authors decompose each emotion vector into (i) a shared component, the shift from the neutral mean to the centroid of all emotion means (equivalently, the average of all emotion vectors), and (ii) an emotion-specific residual from that centroid to the target emotion mean. Experiments on IndexTTS-2 and CosyVoice2 show improved alignment with requested single and mixed emotions over CoCoEmo.

Critical strengths
  1. A useful decomposition of emotion vectors. Each emotion vector splits into a shared "away-from-neutral" part and an emotion-specific residual, and the ablations confirm the two play different roles. This decomposition pinpoints a real failure mode of linear emotion mixing, not just a reparameterization.
  2. Well-controlled experimental design. All steered vectors are rescaled to the same reference norm, so improvements come from reweighting directions rather than injecting larger magnitudes.
  3. Training-free and easy to follow.
Critical weaknesses
  1. Limited novelty. EmoRES can be rewritten as subtracting a scaled mean of the emotion vectors. This is essentially similar to mean-centred activation steering. The contribution is closer to a re-weighting of CoCoEmo with one extra scalar than to a new steering method.
  2. The decomposition depends on the emotion inventory. The centroid is defined over the chosen set E. With three of four emotions being high-arousal, the "shared" direction may largely encode arousal and would change if emotions were added or removed.
  3. The objective metrics lack significance tests despite large variance.
Questions for authors
  1. λr and α are fixed globally, while the input text may itself carry emotional content. Could you report results stratified by whether the text sentiment agrees or conflicts with the target emotion, and discuss whether the steering strength should adapt to the text?
  2. Could you provide acoustic analyses (F0 mean/range, energy, speaking rate) for shared-only, residual-only, CoCoEmo and EmoRES outputs? If the shared component mainly encodes arousal, it should mainly shift F0 and energy, while the residual should separate categories with similar arousal (e.g., angry vs. happy).
  3. How does EmoRES relate to mean-centred steering? Does subtracting the mean emotion vector with a tuned coefficient reproduce EmoRES's gains?
4 comments
AuthorsOpenReview ↗

Part (1/3)

Weakness 1: Novelty

We thank the reviewer for the comment. One of our contributions is the one the reviewer identified in Strength 1: the decomposition “pinpoints a real failure mode of linear emotion mixing, not just a reparameterization.”
While EmoRES can be expressed in terms of the mean emotion vector, it is different from mean-centred steering (Jorgensen et al., 2023) in its construction and how it treats the common component:

Different construction. Mean-centring subtracts the mean of general training activations from a single target-dataset mean, to correct for anisotropy in the activation space. EmoRES instead decomposes contrastive emotion-versus-neutral vectors around the centroid of the emotion-specific means. That separates a neutral-to-emotional displacement from the category-specific contrasts, a structure that mean-centring neither identifies nor uses.

Partial versus full removal of the common component. Mean-centring treats the common component as a nuisance offset and removes it with a fixed unit weight. Our central finding is that this component is functional: it moves speech away from neutral (neutral posterior 36.51% → 7.49% for IndexTTS2). The residual-only condition in Table 3, which is the closest analogue of full removal, lowers TEP and E-SIM on both backbones. EmoRES retains the shared component and strengthens the residual, and this outperforms both vanilla steering and full removal.

Mixed-emotion composition. Mean-centring targets a single behaviour. EmoRES is designed for compound emotions: we show that under convex composition, the shared component’s coefficient stays fixed while the residual alone encodes the requested proportions, which is why strengthening the residual improves adherence to the mixture (ρ 22.00 → 48.13 on held-out IEMOCAP, human Dom-hit +19 points, Fidelity improved significantly).



Weakness 2 and Question 2: Arousal in the shared component

We thank the reviewer for this hypothesis. We simplify this question into two:

  • Does the shared component encode arousal-related effects? The shared component raises F0 and energy on average. Table D shows that shared-only steering on average increases mean F0 and energy on both backbones compared to no steering, consistent with an arousal-related acoustic effect.

Table D. Difference of acoustic changes between steering with only the shared component and unsteered baseline on the CREMA-D evaluation subset.

BackboneF0 mean Δ(Hz)F0 range Δ(Hz)Energy Δ(dB)Rate Δ(words/s)
CosyVoice2+19.68 ± 24.90+31.79 ± 42.11+2.14 ± 2.44+0.006 ± 0.720
IndexTTS-2+16.49 ± 15.30+17.88 ± 51.15+1.29 ± 1.67-0.395 ± 0.336

  • Can the residual separate categories with similar arousal (e.g., angry vs. happy)? Yes. In Table E, we tested whether the residual distinguishes angry from happy when their predicted arousal is similar. Using an independent arousal regressor [R], we selected paired residual-only outputs with an absolute arousal difference ≤0.05, and classified each output as angry or happy with the emotion2vec+ large speech-emotion recognizer. For both backbones, residual-only steering has higher angry and happy recall than shared-only steering. These results support the residual’s role in distinguishing emotion categories under similar predicted arousal.

Table E. Recall for emotions angry and happy on the CREMA-D evaluation subset.

BackboneMethodAngry recall (%)Happy recall (%)
CosyVoice2Shared-only28.2171.79
CosyVoice2Residual-only58.9784.62
IndexTTS-2Shared-only39.8760.13
IndexTTS-2Residual-only66.6780.39

[R] Wagner, Johannes, et al. "Dawn of the transformer era in speech emotion recognition: closing the valence gap." IEEE TPAMI 2023

AuthorsOpenReview ↗

Part (2/3)

Question 1: Effect of text sentiment on steering

Table C shows the performance gains (TEP and E-SIM) of EmoRES compared to CoCoEmo with IndexTTS-2 as backbone for different text-target sentiment relations, agree, conflict, and neutral.

Agrees: The text expresses the requested emotion, such as happy text with a happy target.
Conflicts: The text expresses a different emotion, such as happy text with a sad target.
Neutral text: The text contains no explicit emotional content, while generation still targets a specified emotion.

Table C shows that EmoRES improves mean TEP and E-SIM across all three text–target sentiment relations on IndexTTS-2. The largest gains occur when text and target emotion conflict, supporting residual enhancement’s usefulness for expressing the requested emotion despite competing lexical cues. Positive average gains also remain for agreeing and neutral texts.

Table C. Performance gains of EmoRES compared to CoCoEmo with IndexTTS-2 as backbone for different text-target relations.

Text–target relationΔTEP ↑ΔE-SIM ↑
Agrees+3.19 ± 3.01+1.87 ± 1.49
Conflicts+14.30 ± 3.49+5.81 ± 1.49
Neutral text+10.80 ± 4.59+4.38 ± 2.22

Weakness 3: Significance tests

We updated Table 1 in the revised manuscript and marked the significant cells with a light-grey background. On IEMOCAP, EmoRES outperforms CoCoEmo on all emotion-related metrics significantly.


Question 3: Relation to mean-centred steering

On subtracting the mean emotion vector with a tuned coefficient. Whether this reproduces EmoRES’s gains depends entirely on the coefficient. The standard choice, full subtraction as in mean-centring, corresponds to removing the shared component, and Table 3 shows this lowers TEP and E-SIM on both backbones (residual-only). Subtracting nothing (a coefficient of 0) leaves the original emotion vector, which recovers CoCoEmo. Results show that the gains appear only in the intermediate regime that our decomposition identifies: retaining part of the shared component while strengthening the category-specific residual. Mean-centring provides no reason to look for this regime, since it treats the common component as an offset to remove. Our functional analysis in Table 3 shows why the component should be kept.

AuthorsOpenReview ↗

Part (3/3)

Weakness 2: Dependence on the emotion set

Table P. Shared-only acoustic changes relative to matched no-steer outputs on CREMA-D, using the setting (λc,λr,α)=(1,0,5)(\lambda_{c}, \lambda_r, \alpha) = (1, 0, 5). F0 uses Swift-F0. F0 range is the voiced 95th–5th percentile span.

BackboneCentroidF0 mean Δ (Hz)F0 range Δ (Hz)Energy Δ (dB)Rate Δ (words/s)
CosyVoice2Original four emotions+19.68 ± 24.90+31.79 ± 42.11+2.14 ± 2.44+0.006 ± 0.720
CosyVoice2Omit angry+21.12 ± 24.56+42.62 ± 54.88+2.08 ± 2.98-0.021 ± 0.808
CosyVoice2Omit happy+18.54 ± 27.11+33.78 ± 46.42+1.79 ± 2.53-0.053 ± 0.736
CosyVoice2Omit sad+22.79 ± 23.62+37.53 ± 45.18+2.20 ± 2.47+0.094 ± 0.764
CosyVoice2Omit surprise+16.33 ± 23.72+29.45 ± 47.38+1.92 ± 2.43-0.081 ± 0.702

BackboneCentroidF0 mean Δ (Hz)F0 range Δ (Hz)Energy Δ (dB)Rate Δ (words/s)
IndexTTS-2Original four emotions+16.49 ± 15.30+17.88 ± 51.15+1.29 ± 1.67-0.395 ± 0.336
IndexTTS-2Omit angry+15.71 ± 14.16+17.64 ± 45.74+1.06 ± 1.54-0.372 ± 0.339
IndexTTS-2Omit happy+12.45 ± 13.78+12.04 ± 43.85+0.92 ± 1.63-0.393 ± 0.338
IndexTTS-2Omit sad+17.77 ± 15.52+17.30 ± 41.74+1.57 ± 1.61-0.316 ± 0.356
IndexTTS-2Omit surprise+12.47 ± 15.00+12.65 ± 42.42+0.97 ± 1.69-0.487 ± 0.388

We recomputed the centroid with each of the four emotions omitted in turn and measured the acoustic effect of shared-only steering relative to no steering, as shown in Table P (same settings as Table D). The effect is stable in direction: mean F0, F0 range and energy increase on both backbones under every centroid. Its size depends on the emotion set in the way the reviewer anticipated. Omitting sad, the only low-arousal emotion, gives the largest increases in mean F0 and energy on both backbones, while omitting happy or surprise reduces them. These shifts are small relative to the utterance-level spread.

Overall, the shared component’s acoustic effect holds for every emotion set we tested, and its strength varies moderately with which emotions define the centroid.

Dear Reviewer,

We appreciate the effort you put into reviewing our paper. With the discussion period closing in less than 24 hours, we would welcome your thoughts on whether our responses address the points you raised.

Kind regards,
The Authors

Reviewer_km3sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EmoRES, a training-free activation steering method for emotional TTS. It decomposes emotion steering vectors into a shared component and an emotion-specific residual, allowing their contributions to be controlled independently. Experiments on multiple TTS backbones show improvements over CoCoEmo in emotion control, with additional support from component ablations and human evaluations.

Critical strengths
  1. EmoRES enables independent control of two vector components without retraining the TTS backbone.

  2. The results suggest complementary roles for the shared and residual components, supporting the proposed design.

  3. Experiments cover multiple backbones, emotion recognizers, and human evaluations, with consistent improvements over CoCoEmo.

Critical weaknesses
  1. Although the decomposition is mathematically valid, the interpretation of the shared component as general emotional activation and the residual as emotion-specific information is not fully established. The ablations provide supporting evidence but do not fully validate this functional interpretation.

  2. The target emotion proportions are derived from corpus rater proportions, but their perceptual meaning remains unclear. Rank correlation measures agreement in relative ordering rather than accurate recovery of the target distribution. Further clarification and stronger perceptual validation would strengthen the claims about mixed-emotion control.

  3. The paper investigates the global steering strength and residual coefficient, but fixes the shared-component coefficient at 1 in the main experiments. A more systematic analysis of the interaction between the two coefficients would help establish the robustness of the method.

  4. The paper primarily compares EmoRES with CoCoEmo. A clearer comparison with related emotion-steering approaches, particularly EmoShift[1] and EmoSteer-TTS[2], would help establish the novelty of the proposed method and distinguish component-wise reweighting from existing emotion-intensity control techniques.

[1] EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesis

[2] EmoSteer-TTS: Fine-Grained and Training-Free Emotion-Controllable Text-to-Speech via Activation Steering

Questions for authors
  1. What evidence supports the functional interpretation of the shared and residual components beyond the current ablation results?

  2. How should the target emotion proportions be interpreted perceptually, and how well do human judgments match the requested distributions?

  3. How sensitive is EmoRES to the shared-component coefficient and its interaction with the residual coefficient?

  4. How does EmoRES compare with existing emotion-steering methods under comparable experimental settings?

  5. How reliable is emotion2vec as an evaluation tool? Some studies have raised concerns about its limited generalizability across different datasets and settings[3]. Could this affect the reliability of the reported results?

[3] Emo-LiPO: Listwise Preference Optimization for Fine-Grained Emotion Intensity Control in LLM-based Text-to-Speech

5 comments
AuthorsOpenReview ↗

Part (1/2)

Weakness 1 and Question 1: Functional interpretation

We thank the reviewer for confirming that the decomposition is mathematically valid and that the ablations provide supporting evidence. We address the concerns of functional interpretation.
Part of the interpretation follows from the construction. The shared component is identical for each requested emotion, so the steering vectors for any two emotions differ only in their residuals. The empirical claim is that the shared component moves speech away from neutral. The following evidence supports this:

  • Activation-space analysis. To test the interpretation directly, rather than only through its effect on generated speech, we analyzed the cached CosyVoice2 activations of the CREMA-D source recordings used for vector extraction. Using 5-fold cross-validation, in Table A, we computed the shared and per-emotion residual directions on the training folds, projected the held-out activations onto them, and evaluated:
    • (i) neutral-vs-emotional detection by AUROC (binary classification, chance value 0.5)
    • (ii) emotion-category classification by balanced acc (chance value 33.3% for 3 emotions).

Table A. Activation-space probing of the shared and residual components on CosyVoice2 (CREMA-D).

LayerNeutral vs. emotional AUROC of SharedNeutral vs. emotional AUROC of ResidualCategory acc. (%) of SharedCategory acc. (%) of Residual
140.883±0.0190.675±0.02058.16±3.2278.17±6.03
170.847±0.0190.680±0.01662.11±4.6680.63±4.44

As shown in Table A, at both layers the shared projection better separates neutral from emotional activations (e.g., Layer 14: AUROC 0.883 vs 0.675), while the residual projections better separate emotion categories (e.g., Layer 14: 78.17% vs. 58.16% balanced accuracy). This directly supports the interpretation in activation space: the shared component is strongly associated with the neutral-vs-emotional distinction, and the residual with emotion identity.

  • Behavioral (Table 3): shared-only steering reduces the neutral posterior from 36.51% to 7.49% (IndexTTS-2) and raises E-SIM and TEP over no steering. However, its rank correlation ρ is low (5.30% / 7.20% on IndexTTS-2 / CosyVoice2). ρ is the Spearman correlation between the requested emotion proportions and the change in the SER’s emotion posteriors. A low value means the output doesn't follow the requested emotion mixture. Residual-only steering, by contrast, follows the requested mixture (ρ = 45.16 and 39.94 on IndexTTS-2 and CosyVoice2), but has lower TEP and E-SIM than the combined vector.
  • Representational (Fig. 2): the output of shared-only steering shows no emotion-specific clusters (probe 57.6%, silhouette≈0), whereas the output of residual-only steering forms distinct clusters (probe 81.4%, silhouette 0.379).
  • Direction-specific controls (App. D, Table 5): replacing either component with a magnitude-matched random direction removes its specific effect. Residual randomization collapses ρ (25.24%→5.62% on IndexTTS-2; 32.48%→0.78% on CosyVoice2), and shared randomization lowers E-SIM and TEP (e.g., CosyVoice2 TEP 68.80%→61.63%). The effects thus come from each component’s direction, not energy.
  • Consistency: the behavioral and control results hold on both backbones used for the component ablations (Table 3; App. D Table 5). The resulting gains of EmoRES over CoCoEmo hold across all four backbones (Tables 1, 6, 7) and under three independent SER evaluators (App. H, Table 10).

Weakness 2 and Question 2: Perceptual meaning and validation of target emotion proportions

Perceptual meaning. A target proportion pep_e is the fraction of the raters who labelled the original recording as emotion ee. Our listening test similarly measures the proportion of listeners who assign each emotion label to the generated speech (Eq. 14), allowing direct comparison with the target proportions. A 60% sad / 40% angry target means that the desired listener-label distribution is 60% sad and 40% angry. Our evaluation measures how closely judgments of the generated speech match this distribution. It does not assume that each listener perceives both emotions simultaneously.
Ordering vs. distribution. We agree that ρ captures ordering only, so we pair it with Fidelity: one minus the total variation distance between the listeners’ label distribution and the target (Eq. 16). EmoRES significantly improves Fidelity over CoCoEmo on both backbones (60.05%→66.32%, 50.80%→59.60%; Table 2). This shows closer agreement with the full mixture, not just its ordering. For the objective metrics, ρ ranks posterior increases rather than raw posteriors, because SER posteriors are not calibrated to human proportions and their calibration differs across recognizers (Table 10).
Validation. Our human study covers 690 samples, with at least 9 listeners each, gold-standard quality checks, and paired bootstrap CIs (App. C.4).

AuthorsOpenReview ↗

Part (2/2)

Weakness 3 and Question 3: Sensitivity to the shared-component coefficient

The shared and residual components are weighted by (λc,λrλ_c, λ_r), respectively. Because the composed steering vector is rescaled to a fixed per-layer norm (second paragraph of Section 4.2), its direction depends on the ratio λrλc\frac{λ_r}{λ_c}, rather than the common scale of the coefficients. For example, (λc,λr)=(2,4)(λ_c, λ_r)=(2,4) and (1,2)(1,2) yield identical normalized steering vectors at the same global steering strength. Fixing λc=1λ_c=1 therefore removes redundant combinations of parameters. We also have Table 3 in the original manuscript, which separately covers the λc=0\lambda_c=0 (residual-only) case.


Weakness 4 and Question 4: Comparison with related emotion-steering methods

EmoSteer-TTS is a training-free method that steers activations in the flow-matching module. CoCoEmo steers the speech LM. In Table B (the EmoSteer-TTS results are now Table 4 in the revised paper), we show that EmoRES improves both.

EmoShift instead trains a layer that learns emotion-specific projection matrices that produce steering offsets, with intensity controlled by scaling that vector at inference. EmoRES differs: it requires no training, and rather than learning or scaling a single vector per emotion, it decomposes each emotion vector into a shared component and a category-specific residual and reweights the two independently.

EmoRES changes how an emotion vector is composed, so it can be applied to other steering methods, such as CoCoEmo (speech LM steering) or EmoSteer-TTS (flow-matching steering). By applying EmoRES to EmoSteer-TTS, it improves all four metrics, as it does for CoCoEmo. The combination of EmoRES with EmoSteer-TTS gives the best ρ and H-Rate, and the combination of EmoRES with CoCoEmo gives the best E-SIM and TEP.

When compared with EmoShift, these higher emotion-control scores of EmoRES are obtained without training emotion-specific projection matrices.

Table B. Our proposed EmoRES method compared with existing methods EmoShift and EmoSteer-TTS with CosyVoice2 as backbone on CREMA-D.

MethodE-SIM ↑TEP ↑ρ ↑H-Rate ↑
CoCoEmo74.16 ± 0.0967.43 ± 0.4032.48 ± 2.1778.92 ± 0.54
CoCoEmo + EmoRES (ours)74.71 ± 0.2468.80 ± 0.5651.94 ± 6.3084.76 ± 2.07
EmoSteer-TTS72.17 ± 0.4362.62 ± 0.3846.37 ± 5.4681.62 ± 2.36
EmoSteer-TTS + EmoRES (ours)73.90 ± 0.4465.07 ± 0.3155.92 ± 3.0385.05 ± 1.05
EmoShift59.42 ± 0.2734.83 ± 1.521.32 ± 12.9867.57 ± 4.05

Question 5: Reliability of emotion2vec as an evaluator

We thank the reviewer and agree that SER reliability is an important concern. We note that Emo-LiPO motivates fine-tuning emotion2vec on its own dataset with a brief remark on out-of-domain performance, and continues to use emotion2vec for evaluation. However, it does not report a cross-dataset analysis. Nevertheless, we agree the concern merits checking, and we show that our results do not depend solely on emotion2vec. In the original manuscript, we address this concern in three ways.
(i) Independent recognizers: Table 9 (Table 10 in the revised manuscript) in Appendix H rescores the same generated audio with two recognizers that differ substantially from emotion2vec, Qwen3-Omni and the Odyssey 2024 WavLM model. EmoRES outperforms CoCoEmo on TEP, ρ, and H-Rate in all 18 comparisons, so the conclusions do not depend on emotion2vec.
(ii) Human evaluation: Dom-hit and Fidelity are computed from listener labels without any SER, and both improve significantly on both backbones (Table 2).
(iii) Relative comparison: all methods are scored by the same recognizer on identical text and prompts, and ρ and H-Rate use posterior increases over the unsteered output, so recognizer-specific bias largely cancels in the comparison. We have cited Emo-LiPO and discussed this in the revision of the paper (marked in blue in Appendix H).

Dear Reviewer,

We appreciate the effort you put into reviewing our paper. With the discussion period closing in less than 24 hours, we would welcome your thoughts on whether our responses address the points you raised.

Kind regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal summary

We thank all reviewers for their constructive feedback.

Strengths noted by reviewers

  • EmoRES is simple, training-free and practical (km3s, nBEn, NBRi, jjBU).
  • Norm-matched ablations separate the effect of direction from magnitude (nBEn, NBRi).
  • Gains hold across backbones, held-out data, recognizers and human evaluation (km3s, NBRi, jjBU).

Main concerns and our responses

  • Novelty relative to CoCoEmo (nBEn, jjBU, NBRi): EmoRES changes only how emotion vectors are composed, so it is not tied to CoCoEmo’s pipeline. Applied to EmoSteer-TTS, which steers the flow-matching module instead of the speech LM, it improves all four emotion metrics (new Section 5.4 and Table 4 in revised paper).
  • Functional interpretation (km3s, NBRi, jjBU): New activation-space probing (Table A) shows that the shared direction better separates neutral from emotional speech (Layer 14: AUROC 0.883 vs. 0.675), while the residuals better identify the emotion (Layer 14: 78% vs. 58% balanced accuracy). Shared-only steering raises non-neutral predictions from 52% to 85%, but unevenly across emotions. We therefore describe the shared component as a request-independent shift away from neutral, not as category-neutral, with the residual carrying the requested emotion identity.
  • Dependence on the emotion set (jjBU, nBEn, NBRi): With a 6-emotion centroid, the ρ and H-Rate gains over CoCoEmo hold. With each emotion omitted in turn, shared-only steering still raises F0 and energy. Reviewer jjBU confirmed that this concern is resolved and raised their score.
  • Comparison with other methods (km3s): We compare with EmoSteer-TTS and EmoShift on CosyVoice2 (Table B).
  • Evaluation reliability (km3s, nBEn): We added significance markers to Table 1, and discuss recognizer reliability with three independent recognizers and human evaluation.

Other new analyses

  • Acoustic analysis and an arousal-matched angry/happy test (nBEn).
  • Results stratified by text sentiment (nBEn).
  • Vector stability across speakers and corpora, and a trimmed-mean estimator (NBRi).
  • Per-emotion robustness of the coefficient setting (jjBU).

Changes in the revised paper (text marked in blue)

  • New Section 5.4 and Table 4: EmoRES applied to flow-matching steering.
  • Table 1: significance markers.
  • Appendix H: discussion of recognizer reliability, citing Emo-LiPO.
  • Appendix tables are renumbered by one (former Tables 4–9 are now Tables 5–10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.