- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a TTS system with ability to independently control speaker timbre and style of the synthesized utterance through two separate reference utterances. During training, the proposed method first constructs a style reference by perturbing the timbre, pitch, and content features of the input recording and resynthesizing the result using an off-the-shelf model. The proposed system is then trained to reconstruct the original recording using the input text, the perturbed style reference and a speaker reference. To help the system better distinguish the two references, authors propose branch-aware positional conditioning. The system also uses MeanFlow for faster generation and DPO for post-training.
The evaluation includes comparisons with existing systems, same-backbone representation comparisons, component ablations, and listening tests. The strongest reported result is improved perceived style similarity, with competitive speech quality and speaker similarity.
Critical strengths
-
The proposed work shows substantial improvement in perceived style similarity over established baseline systems. At the same time, the speech quality and timbre similarity remain competitive. Although the evaluation protocols aren't clearly specified, the system shows useful improvement for expressive TTS.
-
The paper contains extensive experiments including component ablations, same-backbone comparisons and supplementary tests of reference processing, reference duration, and cross-language and cross-gender transfer. So it helps understand the contribution of each component beyond the comparison of complete systems.
Critical weaknesses
-
It is unclear what is new beyond prior work. Self-reconstruction with information perturbation has been used in SpeechSplit/SpeechSplit 2.0 and NANSY/NANSY++, and separate style and timbre references are already used in Vevo and several other TTS systems. The authors should explain what their perturbation and resynthesis procedure adds. The motivation also presents limitations of joint reference conditioning as broader limitations of existing controllable TTS methods which is not the case.
-
Important experimental details and results need clarification. The paper does not clearly identify the evaluation data, its overlap with training, or how baseline representations are adapted for the same-backbone comparison. The very high WER values in Tables 3 and 5 are not adequately explained. Tables 5 and 12 also contain matching values under different perturbation labels, despite evaluating different signals. These issues make the results difficult to interpret. Further, an emotion-based similarity score provides limited evidence that perturbation preserves broader speaking styles.
Questions for authors
-
Why do the “without content” and “without timbre” rows in Table 5 match the oppositely labeled rows in Table 12? Please clarify which labels and results are correct.
-
How are the timbre and style references selected during training? Do they overlap with the target recording? Please define “same-utterance” and “non-overlap” in Table 3, and confirm whether each Table 5 variant is retrained. Can you provide transcripts showing what causes the extreme WER values?
-
Where do the evaluation recordings come from, and are their speakers and utterances excluded from training and DPO data? How many samples and ratings are used in each listening test, and how are confidence intervals calculated?
-
What specifically distinguishes the proposed perturbation procedure from prior methods? Is there direct listening-test evidence that the perturbed references preserve the intended style?
-
Please specify the perturbation sampling rules, BAPC positional offsets, MeanFlow training target, and the loss actually used for acoustic DPO training.
8 comments
R3.2 Reference sampling, overlap definitions, model retraining, and high WER
Reference selection. The style reference is constructed from the target utterance itself through attribute perturbation and resynthesis (§3.2):
"rather than requiring manually curated style pairs or style annotations, we explicitly perturb non-style factors in the target speech and resynthesize a reference utterance that preserves the dominant speaking style while suppressing interfering cues."
"we introduce an attribute perturbation strategy that constructs an attribute-purified style reference from each training utterance."
The timbre reference is sampled from other segments of the same speaker according to the speaker ID of the target utterance and is not the same recording as the target. The style reference is derived from the target speech, which is part of our training construction.
Definitions of same-utterance / non-overlap. As described in Section 5.2, these terms concern content overlap in the Vevo2-CS representations in Table 3. In the same-utterance setting, the style representation is extracted from the target utterance itself, so the reference and target fully overlap in content. In the non-overlap setting, the reference is taken from another utterance to remove this content overlap. These terms describe only the relationship between reference and target content; they do not imply different speakers or fully independent recording sources.
The importance of this control is illustrated by Table 3: same-utterance yields a WER of 166.588% and an S-SIM of 0.641; after removing content overlap, WER decreases to 2.823% and S-SIM to 0.495. A high style score accompanied by extremely high WER cannot be directly interpreted as effective style transfer. Our AP representation achieves a WER of 0.941%, a T-SIM of 0.751, and an S-SIM of 0.603 under the same backbone, reflecting joint performance in content accuracy and control over timbre and style.
For Table 5, the w/o content and w/o timbre configurations each have a separately trained model, and the reported metrics are computed on their respective generated speech, rather than obtained by disabling perturbation only at inference. Full is the result of further DPO post-training of Base.
WER can exceed 100% by definition, because insertion errors also contribute to the numerator. The beginning of Appendix C of the original manuscript states:
"WER is macro-averaged over utterances and includes insertion errors; it can therefore exceed 100% when the evaluated speech contains repeated or mismatched content."
Section 5.2 also attributes the high WER under same-utterance content overlap as follows:
"Vevo2-CS (same-utt) achieves apparently strong style attribution but suffers from extremely high WER due to content copying."
For final generated speech, copying incorrect content is a content-accuracy failure even if acoustic naturalness remains high; such high WER therefore cannot be dismissed as unrelated to generation quality. For perturbed references, WER instead diagnoses recoverability of the original content, and these two uses should be distinguished; recognition may also be affected by distortion in the reference audio.
Per-sample transcript examples. The following are four English samples drawn from the perturbed references; WER is computed by comparing Whisper's recognition of the perturbed and resynthesized references against the original text:
| Original text | Recognition result for the perturbed reference | WER (%) |
|---|---|---|
| Make your feedback specific. | You're so poor, my dear. | 125.0 |
| People in Huanglong have a good tradition of reciprocating help generously… | No matter how much I try, he just... There are many ways to deal with it… | 103.1 |
| I don't understand... What time? What are you talking about? | The man appears, but he doesn't want to talk. He doesn't know how to get away. | 160.0 |
| Yeah. Boo! I want my money back! | If he didn't plan to make it clear from the start, why would he have to pull himself together? | 271.4 |
The average WER of these four samples is 164.9%. These examples show substantial substitutions and insertions between the recognized text of the perturbed references and the original text, rather than omissions alone; the 271.4% value does not require repeated output. However, ASR results alone cannot fully distinguish content changes from recognition errors caused by distortion. These are reference-side examples illustrating the recoverability of original content assessed in Table 12; they do not directly explain failures of final generated speech in Tables 3 and 5. In the revision, we will separately label reference-side and generation-side per-sample transcripts in App. C.1 to distinguish these two sources of high WER.
R3.3 Evaluation sources, objective grid, and subjective evaluation protocol
Sources and exclusion. The evaluation recordings consist of high-quality samples manually selected from multiple public speech datasets and are kept separate from both the training data and the DPO data.
Objective grid. 16 style references × 8 timbre references (4 male and 4 female) × 16 texts (8 Chinese and 8 English), yielding 2,048 generation conditions (App. D).
"The controlled swapping grid contains 16 style references, 8 timbre references (4 male and 4 female), and 16 target texts (8 Chinese and 8 English), yielding 16 × 8 × 16 = 2,048 utterances."
Section 4.3 of the original manuscript describes the basic listening-test format:
"We conduct a human evaluation with 24 bilingual Chinese-English participants. Each sample is rated on a 5-point scale for subjective style similarity (SS-SIM), subjective timbre similarity (ST-SIM), and overall quality (S-MOS). We report mean scores with 95% confidence intervals."
Appendix B.3 of the original manuscript describes the grouping procedure:
"We invite 24 bilingual Chinese–English participants to take part in the subjective evaluation. To improve reliability and reduce listening fatigue, the evaluation is conducted in grouped listening sessions. Each session contains 10–12 speech samples, and participants are given 5–10 seconds after each sample to record their scores."
Specifically, each listening test in Tables 2 and 4 uses 300 audio samples per system, evaluated in groups by 24 bilingual Chinese–English listeners, with each audio sample rated by 6 listeners. A five-point scale is used to assess naturalness, timbre similarity, and style similarity separately. The scoring guidelines are provided in Table 10 (style) and Table 11 (timbre) of Appendix B.3 of the original manuscript. Subjective evaluation results are reported as means with 95% confidence intervals, calculated using the normal approximation: , where is the mean rating, is the sample standard deviation, is the number of observations used in the calculation, and is the standard error of the mean (SE). The values following ± in the tables are the confidence-interval half-widths, namely , rather than standard deviations or standard errors themselves. We will add this calculation procedure to the revised manuscript.
Response to Reviewer 9AmC
R3.1 Complete corrections to Tables 5 and 12
Thank you for identifying this issue. We acknowledge reporting errors in the original tables. We have removed the three individual perturbation rows from Table 5 and clarified the perturbation configurations in Table 12 (Appendix C.1 of the original manuscript), distinguishing “only” (applying only the specified perturbation) from “w/o” (removing the specified perturbation). We apologize for the confusion and clarify the evaluation targets below.
Table 5 evaluates the inference outputs of dual-reference models. Each ablation configuration is used to construct style references during training and to train the corresponding dual-reference model. During evaluation, the model generates speech conditioned on the target text, timbre reference, and style reference, and the metrics are computed on the final generated speech. This table examines how each design affects content accuracy, target-timbre preservation, and style transfer. Here, w/o indicates the removal of the corresponding component or perturbation, and all experiments are conducted within the MeanFlow framework. This framework-specific scope is stated in the original caption:
Table 12 (Appendix C.1 of the original manuscript) evaluates input audio reconstructed after perturbation. The object evaluated is stated at the beginning of Appendix C.1 of the original manuscript:
Specifically, it directly measures the retention of timbre, style, and content information in the reconstructed audio relative to the original input, without downstream dual-reference TTS inference. SVC-only involves reconstruction without attribute perturbation; pitch only, content only, and timbre only each enable a single perturbation; Full enables all attribute perturbations. Thus, the two tables evaluate different objects for different purposes, and their values should not correspond one-to-one. The duplicated values and cross-table correspondences in the original manuscript resulted from table-reporting errors and are not evidence that the two experiments produced identical results.
Table 5: Generation results of ablated models under dual-reference conditioning within the MeanFlow framework.
Table 12: Information retained after SVC reconstruction of the same input audio under different perturbation configurations.
In Table 5, a higher T-SIM indicates that the generated speech is closer to the specified target timbre; in Table 12 (Appendix C.1 of the original manuscript), a lower T-SIM indicates weaker cues to the original speaker's identity in the reconstructed reference. WER in Table 12 (Appendix C.1 of the original manuscript) is computed against the original input text; an increase serves to diagnose reduced recoverability of the original content, not improved synthesis quality. In addition, Full in Table 5 denotes the complete model, namely Base with DPO, whereas Full in Table 12 (Appendix C.1 of the original manuscript) only means that all three attribute perturbations are enabled. We will update Section 5.3, Appendix C.1, and the table notes accordingly to make these distinctions explicit.