acceptodds
Under review as a conference paper at ICLR 2027

SEmoEdit: Probing and Harnessing the Editability of Pre-trained Speech Flows

Abstract

Existing training-based speech emotion editing methods often require substantial task-specific training and can be unstable. This motivates us to investigate whether the pretrained generative dynamics of large-scale text-to-speech (TTS) models can be directly manipulated for training-free emotion editing. To answer this question, we probe the editability of pretrained flow-matching and hybrid TTS models by constructing a controlled test set and systematically diagnosing editing effects along the generative trajectory. Our analysis reveals that pretrained TTS models are substantially editable in emotion, but such editability is architecture- and trajectory-dependent and can be disrupted by early flow-matching steps, while cross-speaker emotion transport carries additional acoustic attributes beyond emotion. To address these limitations, we propose **SEmoEdit**, the first training-free framework that formulates emotion editing as dynamic velocity transport between source and target emotions, enabling robust, flow-based speech emotion editing directly within pretrained TTS models. SEmoEdit unifies three core operations: emotion replacement, emotion erasure, and continuous emotion interpolation, requiring neither parameter updates nor task-specific optimization. To systematically evaluate these capabilities, we introduce **SEmoEditBench**, a dataset comprising 600 editing cases, and conduct extensive experiments across state-of-the-art (SOTA) models and backbones. Our results show that SEmoEdit is highly effective and broadly applicable, outperforming existing training-based and activation-steering methods. Ultimately, this work reveals that pretrained speech flows possess rich, latent emotion-editing capabilities, providing useful guidance for real applications. **Code** and **benchmark** are provided in the supplementary material. **Audio samples** are available at https://semoedit.pages.dev/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
emotion editing, text-to-speech, flow matching, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_bxsvOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether pretrained flow-matching and hybrid TTS models can be directly repurposed for training-free speech emotion editing. The authors first analyze the editability of pretrained speech flows and study where along the generative trajectory emotion editing is effective, as well as whether source-to-target velocity transport also carries non-emotional attributes. Based on these observations, the paper proposes SEmoEdit, which repeatedly computes a target-conditioned versus source-conditioned velocity difference during sampling and uses this difference to update an edited speech state. The framework supports emotion replacement, erasure, and intensity control, with additional speaker alignment and an emotion-bridging procedure for interpolation. The paper also introduces SEmoEditBench, a 600-case evaluation benchmark, and evaluates the method on F5-TTS, CosyVoice 2, and IndexTTS2 against training-based and activation-steering baselines. The paper claims that dynamic velocity transport provides stronger emotion editing than existing alternatives. The paper explicitly positions the contribution as the first training-free formulation of speech emotion editing via dynamic velocity transport.

Critical strengths

Clear presentation and broad experimental coverage

The paper is well written and easy to follow. The proposed approach is evaluated across multiple pretrained TTS backbones and emotion-editing tasks. Beyond the main results, the authors provide analyses of trajectory dependence, speaker leakage, and interpolation behavior, offering additional insight into the method.

A useful benchmark and diagnostic setup

SEmoEditBench covers emotion replacement, erasure, and intensity control across different speakers and datasets. This provides a useful testbed for examining emotion-editing behavior in pretrained speech models.

Critical weaknesses

Insufficient evidence isolating the contribution of dynamic velocity transport

The reported improvements are difficult to attribute specifically to the proposed state-dependent velocity transport. The strongest SEmoEdit results depend on explicit target-emotion audio references and speaker-converted target conditioning, while interpolation additionally requires reference retrieval and a second editing pass. The paper also reports a substantial performance drop when audio conditioning is replaced with text conditioning.

These factors make it unclear how much of the improvement comes from dynamic transport rather than the additional target information and preprocessing. A controlled comparison that retains the same target references and preprocessing but replaces dynamic transport with a simpler static transport rule would help isolate its contribution. This is particularly important given the similarity between the proposed source-to-target velocity-difference formulation and existing inversion-free flow-editing approaches.

Limited evidence for speech fidelity beyond emotion-editing performance

The evaluation focuses primarily on emotion-editing success, with less convincing evidence for intelligibility, speaker preservation, and overall speech quality. Several emotion metrics, including E-SIM, DES, and EIC-Emb, rely on emotion2vec representations. Since the interpolation pipeline also uses emotion2vec for bridge-reference retrieval, the intensity-control evaluation is not fully independent of the representation used by the method.

Furthermore, the main comparisons report relative WER/CER without absolute recognition error rates, making it difficult to assess the intelligibility of the edited speech. This concern is reinforced by the interpolation analysis, where the absolute WER/CER reaches 0.603 after bridging for IndexTTS2. SEmoEdit also shows relatively low speaker-similarity and UTMOS scores in several settings.

Taken together, the results demonstrate effective emotion modification, but provide insufficient support for broader claims of robust, high-fidelity emotion editing.

Questions for authors
  1. Can the authors compare static and dynamic transport while keeping the backbone, target reference, speaker alignment, and preprocessing fixed?

  2. How does the proposed transport formulation differ from prior inversion-free flow editing? An equation-level comparison would clarify the methodological novelty.

  3. Can the authors evaluate intensity control using metrics independent of emotion2vec, which is also used for bridge retrieval?

  4. Can the authors report absolute source and edited WER/CER alongside relative changes, particularly given the high errors observed after bridging?

  5. How were the activation-steering baselines tuned, including steering strength, layer/token selection, and backbone-specific settings? Were official implementations used?

5 comments
AuthorsOpenReview ↗

Response to Reviewer bxsv (1/2)

We thank the reviewer for recognizing our broad experimental coverage, trajectory analysis, and benchmark. We conducted additional controlled experiments and independent evaluations to further substantiate our contributions.


W1/Q1 — Contribution of dynamic velocity transport

First, we clarify that our main evaluation (Table 2) uses the same-speaker setting, where speaker alignment reduces to an identity mapping, requiring no voice conversion. Moreover, even with text conditioning (without target audio), SEmoEdit outperforms activation-steering baselines in emotion transfer (TEP 0.274/0.354 vs. 0.025–0.082).

To directly isolate dynamic transport, we conducted a controlled static-vs-dynamic comparison on 94 speaker-disjoint inputs per backbone, keeping source representations, target references, preprocessing, noise draws, decoders, and branch-query budgets identical. Static transport fixes the velocity-difference direction at a calibration-selected timestep. Neither variant uses additional voice conversion or emotion bridging.

At α=1.0, results are reported as Static / Dynamic:

BackboneTEP ↑S-SIM ↑EN WER ↓
F5-TTS0.160 / 0.5090.528 / 0.3740.267 / 0.115
CosyVoice 20.209 / 0.5620.338 / 0.3862.643 / 0.116
IndexTTS20.269 / 0.7070.381 / 0.4280.251 / 0.086

Dynamic transport substantially improves emotion transfer and English intelligibility across all three backbones, while also improving speaker preservation for both hybrid backbones. These results establish its contribution independently of reference information and preprocessing.

Revision: Sec. 4.3, Table 5, and Appendix E.4 provide results at α=0.5/1.0 and detailed protocols.


W1/Q2 — Relationship to FlowEdit in image domain and methodological novelty

Our formulation builds on FlowEdit, as explicitly acknowledged in Sec. 3.1. Under the time transformation s=1−ts=1-t and v(q,t;c)=−V(q,1−t;c)v(q,t;c)=-V(q,1-t;c), its update becomes:

qs=(1−t)ϵ+tx,qt=qs+Et−x,q_s=(1-t)\epsilon+tx,\quad q_t=q_s+E_t-x,

E˙t=Eϵ[v(qt,t;ctgt)−v(qs,t;csrc)],\dot E_t=\mathbb{E}_{\epsilon}[v(q_t,t;c_{\rm tgt})-v(q_s,t;c_{\rm src})],

which corresponds to our formulation at α=1,τ=0\alpha=1,\tau=0.

Our contribution lies in systematically investigating and enabling inversion-free editing for pretrained speech models, rather than proposing a fundamentally new ODE. Specifically, we identify and address three speech-specific challenges:

  1. Temporal drift: Skipping 4/32 early steps reduces onset drift from 308 to 97 ms while retaining 99.5% of emotion gain.
  2. Speaker entanglement and prefix handling: Speaker-aligned conditioning and backbone-specific prefix processing enable compatible emotion transport.
  3. Intermediate-state distortion: Emotion bridging enables cleaner intensity-controlled outputs.

Together, these findings establish practical training-free emotion editing across distinct speech architectures.


W2/Q3 — Independent intensity-control evaluation

We added an evaluation using the fixed SUPERB wav2vec2 emotion recognizer, independent of emotion2vec and unused in generation, retrieval, or tuning. All eight systems were evaluated on 90 common RAVDESS/CREMA-D curves per system. We separately measure monotonic ordering (M↑) and endpoint distance to the target emotion distribution (TV↓).

SystemOrdering M ↑Endpoint TV ↓
SEmoEdit / F50.18890.4744
SEmoEdit / Cosy0.04220.3952
SEmoEdit / Index0.15560.4104
CoCoEmo / Cosy0.04440.5018
CoCoEmo / Index0.02670.5036

SEmoEdit achieves better target proximity than CoCoEmo on both backbones, with a statistically supported improvement for CosyVoice 2. Although monotonic ordering is more mixed, independent human evaluation also favors SEmoEdit (EIC-MOS 2.80–3.65 vs. 1.65 for CoCoEmo).

These complementary evaluations support improved target-emotion control beyond emotion2vec-based metrics.

Revision: Sec. 4.2, Appendix E.2, and Tables 22–23 provide all eight systems, endpoint accuracy, and confidence intervals.

AuthorsOpenReview ↗

Response to Reviewer bxsv (2/2)

W2/Q4 — Intelligibility, the reported 0.603 WER, and speech fidelity

We identified an English normalization error that penalized equivalent transcriptions (e.g., "eleven" vs. "11"). We corrected this using Whisper's standard English normalizer across all 11 systems, without changing audio or ASR predictions.

Crucially, the reviewer-cited 0.603 WER decreases to 0.000 on the same 450 IndexTTS2 bridged outputs. The main intensity-control source/edited WER similarly changes from 20.00%/61.11% to 0%/0%. These subsets contain only three distinct short sentences, so zero WER does not imply perfect perceptual fidelity.

We additionally report absolute source/edited WER/CER. Across three backbones, corrected English WER decreases from 16.35% to 7.53–11.25% for replacement and from 12.64% to 3.57–6.22% for erasure.

Regarding preservation, our matched-TEP evaluation demonstrates higher S-SIM than dots.tts.edit (Table 6). Human evaluation further shows competitive naturalness (IndexTTS2 N-MOS 3.88/4.15) alongside strong emotion-editing performance (ES-MOS 4.22/4.40). CosyVoice 2/IndexTTS2 replacement UTMOS also remains close to CoCoEmo (gaps of 0.077/0.081).

These corrected and complementary results strengthen the evidence for intelligibility and competitive speech quality while accounting for emotion-transfer strength.

Revision: Secs. 4.1–4.3, Tables 2–4 and 6, and Appendix E.3–E.4 provide the corrected scores, absolute error rates, and preservation analyses.


Q5 — Baseline implementations and tuning fairness

CoCoEmo uses the authors' official implementation and steering vectors, retaining their selected layers and default strength 3 without test-set retuning. Our EmoSteer-inspired comparison uses a mean-pooled activation-steering adaptation, with its differences from the original method explicitly documented.

We further evaluated CoCoEmo strengths 0–5 on 24 common inputs (5 calibration / 19 evaluation). Across this predefined sweep, the maximum observed CoCoEmo TEP on the held-out evaluation inputs is 0.176/0.211 for CosyVoice 2/IndexTTS2, compared with 0.637/0.724 for SEmoEdit on the same inputs.

SEmoEdit's emotion-transfer advantage therefore persists beyond CoCoEmo's default strength, supporting the robustness of our comparison across the tested settings.

Revision: Appendix E.5 and Table 30 provide implementation details, layer/token settings, and the complete strength sweep.

Dear Reviewer bxsv,

As the discussion period draws to a close, we would greatly appreciate it if you could revisit our responses and updated results, and share your thoughts on whether they have adequately addressed your original concerns.

Thank you for your time and consideration.

Reviewer_aFbeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether pretrained TTS models can edit speech emotion without further training. The proposed method, SEmoEdit, updates the source speech using velocity differences between source and target conditions, with speaker conversion and a second editing pass to improve intermediate outputs. The authors evaluate emotion replacement, erasure, and intensity control on a 600-case benchmark using three TTS backbones. The results show better emotion-editing scores than the tested baselines, but speaker preservation and intelligibility remain concerns, especially for intensity control.

Critical strengths

S1: The delayed-start and early-stop experiments show different behavior across backbones and help explain when editing introduces timing shifts or transcription errors.

S2: The method supports three editing tasks without updating model parameters and improves the reported emotion-transfer metrics across all three backbones in the same-speaker setting.

Critical weaknesses

W1: Preservation remains a problem, with replacement S-SIM of 0.372–0.423 compared with Step-Audio-EditX’s 0.567, and a 41.1-percentage-point WER increase for IndexTTS2 intensity control (Table 2).

W2: EIC-Emb scores projected progress [0, 1, 2, 3, 4] at 1.0 but [0, 0.25, 0.5, 0.75, 1] at 0.5, meaning it can reward overshooting the target and does not by itself establish good interpolation (Appendix C.2.1).

Questions for authors

Q1: When methods are compared at similar emotion-transfer scores, does SEmoEdit still offer better speaker preservation and intelligibility?

Q2: Would the intensity-control ranking change if monotonicity and distance to the target were evaluated separately?

3 comments
AuthorsOpenReview ↗

Response to Reviewer aFbe

We thank the reviewer for recognizing our trajectory analysis and training-free editing results. We address your concerns as follows:


W1/Q1 — Preservation at comparable emotion transfer

The original comparison involves substantially different emotion-transfer strengths: SEmoEdit achieves TEP of 0.498/0.554/0.691 across F5/Cosy/Index, compared with 0.222 for Step-Audio-EditX. To ensure a fair comparison, we conducted same-input TEP matching on 94 speaker-disjoint evaluation inputs per backbone (|ΔTEP| ≤ 0.05). Strength candidates were fixed using 26 separate calibration cases, without using preservation metrics for matching.

Results are reported as SEmoEdit / comparator.

BackboneComparatorTEP ↑S-SIM ↑EN WER ↓
F5-TTSStep-Audio-EditX0.171 / 0.1700.529 / 0.55010.87 / 4.15
CosyVoice 2Step-Audio-EditX0.202 / 0.2010.547 / 0.5576.95 / 9.87
IndexTTS2Step-Audio-EditX0.187 / 0.1850.568 / 0.5667.72 / 8.52
F5-TTSdots.tts.edit0.345 / 0.3450.483 / 0.26710.21 / 4.26
CosyVoice 2dots.tts.edit0.374 / 0.3730.522 / 0.2656.78 / 4.14
IndexTTS2dots.tts.edit0.422 / 0.4210.536 / 0.2676.84 / 4.56

At matched emotion transfer, SEmoEdit achieves substantially higher S-SIM than dots.tts.edit and comparable similarity to Step-Audio-EditX, while intelligibility remains method-dependent. These results support competitive preservation alongside stronger training-free emotion editing.

Revision location: Sec. 4.3, “Preservation at similar emotion transfer,” and Table 6 report the comparison. Appendix E.4 and Tables 28–29 specify the candidate strengths, bridging and matching protocols, Chinese CER, and higher-TEP sensitivity results.


W1 — Correction of the reported +41.1-point WER increase

We identified an English normalization error that incorrectly penalized equivalent transcriptions (e.g., “eleven” vs. “11”). We corrected this using Whisper's standard English normalizer, consistently applied to references and ASR hypotheses across all 11 systems, without changing audio or ASR predictions.

On the same 30 intensity-control cases at α=1, IndexTTS2 source/edited WER changes from 20.00%/61.11% to 0.00%/0.00%. Thus, the reported +41.1-point increase was a scoring artifact rather than an observed degradation under corrected scoring.

This resolves the specific WER discrepancy identified by the reviewer. We have recalculated the affected results for all systems.

Revision location: Sec. 4.1 (“Metrics”) specifies text normalization; Sec. 4.2 and Table 2 report corrected absolute error rates alongside their changes. Appendix E.3 and Tables 24–27 provide source/edited WER and CER separately by language, sample counts, and the scoring audit. The recognition columns of the complete benchmark tables in Appendix D.1 are also corrected.


W2/Q2 — Overshooting and separate interpolation metrics

We agree that EIC-Emb's unbounded projection can reward overshooting. To test its impact, we recomputed EIC-Emb after capping projected progress at 1, removing additional credit for overshooting. We also separately evaluated monotonic ordering (M↑) and normalized distance to the target (D↓) on the same 30 intensity-control cases per system.

SystemCapped EIC-Emb ↑Ordering M ↑Target distance D ↓
SEmoEdit / F50.16840.42670.7709
SEmoEdit / Cosy0.16800.40000.7263
SEmoEdit / Index0.18870.44000.7003
CoCoEmo / Cosy0.04070.13330.9312
CoCoEmo / Index0.01700.12000.9939
EmoSteer-TTS / F50.01270.21330.9855
EmoSteer-TTS / Cosy−0.0088−0.04001.0304
EmoSteer-TTS / Index−0.0049−0.26671.0227

The EIC-Emb ranking remains unchanged after removing overshoot rewards, with SEmoEdit consistently outperforming the steering baselines. Importantly, SEmoEdit also achieves better M and D point estimates than all same-backbone steering comparisons. Speaker-cluster bootstrap intervals support all five target-distance improvements, while the F5 monotonicity improvement remains statistically uncertain.

These evaluations demonstrate that SEmoEdit's intensity-control advantage is not driven by overshooting and extends beyond the original EIC-Emb metric.

Revision location: Sec. 4.2, “Comparison with activation steering methods,” now summarizes the overshoot-controlled evaluation. Appendix E.1 defines the separate ordering and target-distance metrics in Eq. (23), reports observed projection ranges and overshoot frequencies, and presents the capped-EIC results and paired confidence intervals in Tables 20–21.

Dear Reviewer aFbe,

As the discussion period draws to a close, we would greatly appreciate it if you could revisit our responses and updated results, and share your thoughts on whether they have adequately addressed your original concerns.

Thank you for your time and consideration.

Dear Reviewer aFbe,

As the discussion period approaches its end, we would sincerely appreciate it if you could revisit our responses and revised manuscript.

We have conducted targeted experiments to directly address your concerns, including emotion-transfer-matched preservation comparisons (Q1) and separate monotonicity and target-distance evaluations (Q2). We have also corrected the WER normalization issue and verified that the intensity-control advantage persists after removing overshoot rewards.

We believe these additional results substantially strengthen the evidence for our contributions and would greatly appreciate your updated assessment.

Thank you again for your time and constructive feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response: Scientific Significance and Key Contributions of SEmoEdit

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their constructive feedback. We also appreciate Reviewer bxsv's acknowledgment of our additional experiments and subsequent rating increase. We have carefully revised the manuscript and conducted new experiments to strengthen our empirical evidence.

Beyond addressing individual concerns, we would like to emphasize the broader scientific significance of this work and its contributions to training-free speech editing.


1. Opening a New Direction: Inversion-Free Editing for Pretrained Speech Models

The central question of our work is: Can the generative dynamics of pretrained speech models be directly harnessed for emotion editing without retraining?

Inspired by inversion-free image editing, we systematically investigate this largely unexplored capability in pretrained speech flows. Importantly, our contribution is not a new mathematical transport equation, but rather establishing how inversion-free editing can be made effective in the speech domain, where temporal alignment, speaker identity, and acoustic fidelity introduce distinct challenges.

Our study goes beyond directly transferring an image-editing technique: it investigates whether, when, and under what conditions pretrained speech models can be edited, and develops a unified training-free framework for emotion replacement, erasure, and intensity control across different TTS architectures.


2. Systematic Findings and Generalizable Research Takeaways

A central contribution of SEmoEdit is the systematic characterization of speech editability, yielding three important findings:

  • Trajectory-dependent editability: Emotion-editing effectiveness varies substantially across generative steps. Skipping early steps reduces onset drift from 308 to 97 ms while retaining 99.5% of emotion gain, highlighting the importance of temporal-aware editing.
  • Attribute entanglement: Source-to-target velocity transport carries speaker identity and pitch characteristics alongside emotion, revealing the need for speaker-aware conditioning and better disentanglement.
  • Intermediate-state limitations: Intermediate transport states do not necessarily correspond to valid speech, motivating emotion bridging and more effective methods for continuous control.
  • Future model design: The gap between audio- and text-conditioned editing highlights the need for stronger instruction-conditioned TTS models and more controllable acoustic representations.

These findings provide actionable guidance for future research, including trajectory-aware generative editing, disentangled acoustic control, and architecture-aware preservation. They also help explain why different pretrained speech architectures exhibit distinct editing behaviors.


3. Additional Evidence Strengthening Our Conclusions

Our rebuttal introduced targeted experiments addressing the reviewers' main concerns:

  • Isolating dynamic transport: Under identical references and preprocessing, dynamic transport improves emotion-transfer performance by 2.6–3.2× over static transport across three backbones (Table 5).
  • Validating preservation and intensity control: Matched-transfer comparisons demonstrate competitive speaker preservation (Table 6). Independent emotion recognition and overshoot-controlled evaluations further support our intensity-control results (Tables 20–23), consistent with human evaluations.
  • Correcting intelligibility evaluation: We identified and corrected an English WER normalization error consistently across all 11 systems, resolving the reported 0.603 WER artifact and providing corrected absolute recognition-error rates (Tables 2 and 24–27).
  • Ensuring baseline fairness: Additional CoCoEmo strength sweeps confirm that SEmoEdit's emotion-transfer advantage persists across the tested configurations (Table 30).

These additions substantially strengthen the empirical foundation of our original findings.


Closing Remarks

We believe the significance of SEmoEdit extends beyond achieving strong emotion-editing performance. More fundamentally, it establishes the feasibility of inversion-free editing in pretrained speech models, systematically reveals the challenges and opportunities unique to speech generation, and provides concrete methodological insights for future training-free editing research.

Together with SEmoEditBench, our 600-case benchmark spanning three editing tasks and diverse speaker/domain settings, we hope this work serves as a useful foundation for further investigation into the controllability and editability of pretrained generative speech models.

We respectfully ask the Area Chair to consider these broader scientific contributions, together with the strengthened experimental evidence, in the final assessment.

We will also open source everything we built.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.