acceptodds
Under review as a conference paper at ICLR 2027

DualRef-TTS: A Label-Free Dual-Reference Framework for Independent Style-Timbre Control in Zero-Shot Speech Synthesis

Abstract

Zero-shot text-to-speech (TTS) systems have achieved significant progress in both speaker similarity and speech naturalness. Building on these advances, recent work has further explored more independent control over emotion and speaking style through a variety of paradigms. However, most existing approaches still rely on human-defined style taxonomies, which limits their ability to achieve reliable dual-reference control, scalable label-free learning, and robust style control for in-the-wild scenarios. We introduce DualRef-TTS, a label-free dual-reference paradigm for independent style–timbre control in zero-shot speech synthesis, where style is learned as a continuous, reference-driven, and speaker-transferable variable, rather than as a predefined label, abstract control vector, or prompt-derived description. First, attribute perturbation constructs attribute-purified style references for label-free style supervision. Second, Branch-Aware Positional Conditioning (BAPC) combines branch embeddings with branch-wise positional offsets to distinguish the timbre reference and style reference, thereby reducing cross-branch interference. Building on this architecture, we further adopt a mean-flow-based acoustic generation method to reduce inference steps while preserving synthesis quality. Finally, Direct Preference Optimization (DPO)-style post-training further aligns the generation distribution with style alignment, semantic accuracy, and prosodic naturalness. Experimental results demonstrate that the proposed method is comparable to state-of-the-art (SOTA) systems across standard metrics, while introducing a label-free dual-reference paradigm that avoids intermediate representations and prompt-based control. This supports more flexible and independent control of timbre and style in open-domain speech synthesis. Audio samples are available at: https://dualref-tts.github.io/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Zero-Shot Text-to-Speech, Style-Controllable Speech Synthesis, Speech Disentanglement, Reference-Based Speech Synthesis, Style–Timbre Control, Flow Matching

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9AmCOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a TTS system with ability to independently control speaker timbre and style of the synthesized utterance through two separate reference utterances. During training, the proposed method first constructs a style reference by perturbing the timbre, pitch, and content features of the input recording and resynthesizing the result using an off-the-shelf model. The proposed system is then trained to reconstruct the original recording using the input text, the perturbed style reference and a speaker reference. To help the system better distinguish the two references, authors propose branch-aware positional conditioning. The system also uses MeanFlow for faster generation and DPO for post-training.

The evaluation includes comparisons with existing systems, same-backbone representation comparisons, component ablations, and listening tests. The strongest reported result is improved perceived style similarity, with competitive speech quality and speaker similarity.

Critical strengths
  • The proposed work shows substantial improvement in perceived style similarity over established baseline systems. At the same time, the speech quality and timbre similarity remain competitive. Although the evaluation protocols aren't clearly specified, the system shows useful improvement for expressive TTS.

  • The paper contains extensive experiments including component ablations, same-backbone comparisons and supplementary tests of reference processing, reference duration, and cross-language and cross-gender transfer. So it helps understand the contribution of each component beyond the comparison of complete systems.

Critical weaknesses
  • It is unclear what is new beyond prior work. Self-reconstruction with information perturbation has been used in SpeechSplit/SpeechSplit 2.0 and NANSY/NANSY++, and separate style and timbre references are already used in Vevo and several other TTS systems. The authors should explain what their perturbation and resynthesis procedure adds. The motivation also presents limitations of joint reference conditioning as broader limitations of existing controllable TTS methods which is not the case.

  • Important experimental details and results need clarification. The paper does not clearly identify the evaluation data, its overlap with training, or how baseline representations are adapted for the same-backbone comparison. The very high WER values in Tables 3 and 5 are not adequately explained. Tables 5 and 12 also contain matching values under different perturbation labels, despite evaluating different signals. These issues make the results difficult to interpret. Further, an emotion-based similarity score provides limited evidence that perturbation preserves broader speaking styles.

Questions for authors
  • Why do the “without content” and “without timbre” rows in Table 5 match the oppositely labeled rows in Table 12? Please clarify which labels and results are correct.

  • How are the timbre and style references selected during training? Do they overlap with the target recording? Please define “same-utterance” and “non-overlap” in Table 3, and confirm whether each Table 5 variant is retrained. Can you provide transcripts showing what causes the extreme WER values?

  • Where do the evaluation recordings come from, and are their speakers and utterances excluded from training and DPO data? How many samples and ratings are used in each listening test, and how are confidence intervals calculated?

  • What specifically distinguishes the proposed perturbation procedure from prior methods? Is there direct listening-test evidence that the perturbed references preserve the intended style?

  • Please specify the perturbation sampling rules, BAPC positional offsets, MeanFlow training target, and the loss actually used for acoustic DPO training.

8 comments

Response to Reviewer 9AmC

R3.1 Complete corrections to Tables 5 and 12

Thank you for identifying this issue. We acknowledge reporting errors in the original tables. We have removed the three individual perturbation rows from Table 5 and clarified the perturbation configurations in Table 12 (Appendix C.1 of the original manuscript), distinguishing “only” (applying only the specified perturbation) from “w/o” (removing the specified perturbation). We apologize for the confusion and clarify the evaluation targets below.

Table 5 evaluates the inference outputs of dual-reference models. Each ablation configuration is used to construct style references during training and to train the corresponding dual-reference model. During evaluation, the model generates speech conditioned on the target text, timbre reference, and style reference, and the metrics are computed on the final generated speech. This table examines how each design affects content accuracy, target-timbre preservation, and style transfer. Here, w/o indicates the removal of the corresponding component or perturbation, and all experiments are conducted within the MeanFlow framework. This framework-specific scope is stated in the original caption:

"Table 5: Ablation analysis of key components in the MeanFlow-based framework."

Table 12 (Appendix C.1 of the original manuscript) evaluates input audio reconstructed after perturbation. The object evaluated is stated at the beginning of Appendix C.1 of the original manuscript:

"We randomly sample 1,000 training utterances and evaluate the processed style references themselves, without involving the generation model."

"Table 12 measures residual speaker identity using T-SIM, retained style using S-SIM, residual linguistic content using WER, and signal quality using DNSMOS."

Specifically, it directly measures the retention of timbre, style, and content information in the reconstructed audio relative to the original input, without downstream dual-reference TTS inference. SVC-only involves reconstruction without attribute perturbation; pitch only, content only, and timbre only each enable a single perturbation; Full enables all attribute perturbations. Thus, the two tables evaluate different objects for different purposes, and their values should not correspond one-to-one. The duplicated values and cross-table correspondences in the original manuscript resulted from table-reporting errors and are not evidence that the two experiments produced identical results.

Table 5: Generation results of ablated models under dual-reference conditioning within the MeanFlow framework.

Model variantWER (%) ↓DNSMOS ↑T-SIM ↑S-SIM ↑
Base1.4833.9110.7130.598
Base w/o BAPC2.0843.8970.6640.633
Base w/o perturbations129.4613.9530.2130.893
Full (Base + DPO)0.9843.9460.7360.624

Table 12: Information retained after SVC reconstruction of the same input audio under different perturbation configurations.

Reference processing conditionT-SIM ↓S-SIM ↑WER (%)DNSMOS ↑
GT——5.23.59
SVC-only (no perturbation)0.8500.9105.93.58
Pitch only0.6550.68510.93.35
Content only0.6500.622120.43.25
Timbre only0.2910.6958.13.53
w/o pitch perturbation0.2240.524115.23.16
w/o content perturbation0.2230.58113.13.35
w/o timbre perturbation0.5040.529172.53.08
Full (all perturbations)0.1740.506114.03.02

In Table 5, a higher T-SIM indicates that the generated speech is closer to the specified target timbre; in Table 12 (Appendix C.1 of the original manuscript), a lower T-SIM indicates weaker cues to the original speaker's identity in the reconstructed reference. WER in Table 12 (Appendix C.1 of the original manuscript) is computed against the original input text; an increase serves to diagnose reduced recoverability of the original content, not improved synthesis quality. In addition, Full in Table 5 denotes the complete model, namely Base with DPO, whereas Full in Table 12 (Appendix C.1 of the original manuscript) only means that all three attribute perturbations are enabled. We will update Section 5.3, Appendix C.1, and the table notes accordingly to make these distinctions explicit.

R3.2 Reference sampling, overlap definitions, model retraining, and high WER

Reference selection. The style reference is constructed from the target utterance itself through attribute perturbation and resynthesis (§3.2):

"rather than requiring manually curated style pairs or style annotations, we explicitly perturb non-style factors in the target speech and resynthesize a reference utterance that preserves the dominant speaking style while suppressing interfering cues."

"we introduce an attribute perturbation strategy that constructs an attribute-purified style reference from each training utterance."

The timbre reference is sampled from other segments of the same speaker according to the speaker ID of the target utterance and is not the same recording as the target. The style reference is derived from the target speech, which is part of our training construction.

Definitions of same-utterance / non-overlap. As described in Section 5.2, these terms concern content overlap in the Vevo2-CS representations in Table 3. In the same-utterance setting, the style representation is extracted from the target utterance itself, so the reference and target fully overlap in content. In the non-overlap setting, the reference is taken from another utterance to remove this content overlap. These terms describe only the relationship between reference and target content; they do not imply different speakers or fully independent recording sources.

The importance of this control is illustrated by Table 3: same-utterance yields a WER of 166.588% and an S-SIM of 0.641; after removing content overlap, WER decreases to 2.823% and S-SIM to 0.495. A high style score accompanied by extremely high WER cannot be directly interpreted as effective style transfer. Our AP representation achieves a WER of 0.941%, a T-SIM of 0.751, and an S-SIM of 0.603 under the same backbone, reflecting joint performance in content accuracy and control over timbre and style.

For Table 5, the w/o content and w/o timbre configurations each have a separately trained model, and the reported metrics are computed on their respective generated speech, rather than obtained by disabling perturbation only at inference. Full is the result of further DPO post-training of Base.

WER can exceed 100% by definition, because insertion errors also contribute to the numerator. The beginning of Appendix C of the original manuscript states:

"WER is macro-averaged over utterances and includes insertion errors; it can therefore exceed 100% when the evaluated speech contains repeated or mismatched content."

Section 5.2 also attributes the high WER under same-utterance content overlap as follows:

"Vevo2-CS (same-utt) achieves apparently strong style attribution but suffers from extremely high WER due to content copying."

For final generated speech, copying incorrect content is a content-accuracy failure even if acoustic naturalness remains high; such high WER therefore cannot be dismissed as unrelated to generation quality. For perturbed references, WER instead diagnoses recoverability of the original content, and these two uses should be distinguished; recognition may also be affected by distortion in the reference audio.

Per-sample transcript examples. The following are four English samples drawn from the perturbed references; WER is computed by comparing Whisper's recognition of the perturbed and resynthesized references against the original text:

Original textRecognition result for the perturbed referenceWER (%)
Make your feedback specific.You're so poor, my dear.125.0
People in Huanglong have a good tradition of reciprocating help generously…No matter how much I try, he just... There are many ways to deal with it…103.1
I don't understand... What time? What are you talking about?The man appears, but he doesn't want to talk. He doesn't know how to get away.160.0
Yeah. Boo! I want my money back!If he didn't plan to make it clear from the start, why would he have to pull himself together?271.4

The average WER of these four samples is 164.9%. These examples show substantial substitutions and insertions between the recognized text of the perturbed references and the original text, rather than omissions alone; the 271.4% value does not require repeated output. However, ASR results alone cannot fully distinguish content changes from recognition errors caused by distortion. These are reference-side examples illustrating the recoverability of original content assessed in Table 12; they do not directly explain failures of final generated speech in Tables 3 and 5. In the revision, we will separately label reference-side and generation-side per-sample transcripts in App. C.1 to distinguish these two sources of high WER.

R3.3 Evaluation sources, objective grid, and subjective evaluation protocol

Sources and exclusion. The evaluation recordings consist of high-quality samples manually selected from multiple public speech datasets and are kept separate from both the training data and the DPO data.

Objective grid. 16 style references × 8 timbre references (4 male and 4 female) × 16 texts (8 Chinese and 8 English), yielding 2,048 generation conditions (App. D).

"The controlled swapping grid contains 16 style references, 8 timbre references (4 male and 4 female), and 16 target texts (8 Chinese and 8 English), yielding 16 × 8 × 16 = 2,048 utterances."

Section 4.3 of the original manuscript describes the basic listening-test format:

"We conduct a human evaluation with 24 bilingual Chinese-English participants. Each sample is rated on a 5-point scale for subjective style similarity (SS-SIM), subjective timbre similarity (ST-SIM), and overall quality (S-MOS). We report mean scores with 95% confidence intervals."

Appendix B.3 of the original manuscript describes the grouping procedure:

"We invite 24 bilingual Chinese–English participants to take part in the subjective evaluation. To improve reliability and reduce listening fatigue, the evaluation is conducted in grouped listening sessions. Each session contains 10–12 speech samples, and participants are given 5–10 seconds after each sample to record their scores."

Specifically, each listening test in Tables 2 and 4 uses 300 audio samples per system, evaluated in groups by 24 bilingual Chinese–English listeners, with each audio sample rated by 6 listeners. A five-point scale is used to assess naturalness, timbre similarity, and style similarity separately. The scoring guidelines are provided in Table 10 (style) and Table 11 (timbre) of Appendix B.3 of the original manuscript. Subjective evaluation results are reported as means with 95% confidence intervals, calculated using the normal approximation: xˉ±1.96×s/n\bar{x} \pm 1.96 \times s / \sqrt{n}, where xˉ\bar{x} is the mean rating, ss is the sample standard deviation, nn is the number of observations used in the calculation, and s/ns/\sqrt{n} is the standard error of the mean (SE). The values following ± in the tables are the confidence-interval half-widths, namely 1.96×SE1.96 \times \mathrm{SE}, rather than standard deviations or standard errors themselves. We will add this calculation procedure to the revised manuscript.

Reviewer_2Uc6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a TTS model called DualRef-TTS that can accept 2 separate inputs: a timbre reference clip and a style reference clip, so that one speaker's voice can be combined with the speaking style of another. During training, the style reference is synthetically constructed by perturbing the target utterance's timbre, pitch and content and resynthesizing it using a singing voice conversion (SVC) model. The approach also contains other components as contributions: per-branch positional embeddings, a final DPO post-training stage, and mean-flow modelling for 4-step inference-time generation. The model is trained on 6000 hours of Chinese and English speech and compared with six baselines. The paper reports the best style similarity in both objective and listening tests, and is close to the best baselines on WER, DNSMOS and timbre similarity.

Critical strengths
  1. The paper presents a simple and practical approach for training dual-reference (timbre + style) TTS without needing style labels or paired data, by synthetically creating the style reference from the target utterance. Separate speech reference clips for timbre and style are useful when the target style is subtle or hard to describe via text. This matters for my assessment because it is the main technical idea and the part most likely to be useful by readers.
  2. Although I don't fully trust the experimental setup (Critical Weakness 2), the results look good, especially on style similarity via the listening tests. The appendix also adds useful checks on reference length, cross-gender and cross-lingual references, and style changes within a reference. If the setup problems are fixed, this would be the strongest evidence that the method works.
Critical weaknesses
  1. The contributions are much less novel than claimed. Without this, what remains is mainly a training recipe, which does not meet the bar in its current state.

    1. Contribution 1: Controlling style using a reference speech clip, separately from the speaker reference clip, goes back to early expressive TTS (e.g. https://arxiv.org/abs/1803.09047, https://arxiv.org/abs/1904.02373), and the paper's own Table 7 lists this very setup for StyleTTS2, Vevo and IndexTTS2. How is this a core contribution of this paper?
    2. Contribution 2: Attribute perturbation is close to the cited NANSY and Gengembre et al. (2024) and to https://arxiv.org/abs/2409.17364 (not cited).
    3. Contribution 3: BAPC is just BERT-style segment embeddings plus position offsets.

    So Contributions 2 and 3 are quite minor as well. The authors should massively narrow their claims and add a simple cascade baseline (TTS prompted with the style clip, then voice conversion).

  2. The experimental setup is not properly described and does not reliably show that DualRef-TTS has better independent (style vs timbre) control than existing systems. The paper's main claims rest on these experiments, and fixing them will need new experiments and results.

    1. It is unclear how most baselines even received both a timbre and a style input (the released Qwen3-TTS models cannot do both).
    2. The test sets are very small (16 style references, 8 speakers).
    3. S-SIM only measures emotion, although the paper defines style more broadly. Also, what model was used for S-SIM is not clear.
    4. Two rows of the Table 5 ablation duplicate Table 12 with swapped labels, and their DNSMOS (3.07 and 3.35) matches the perturbed references rather than generated speech, which makes me feel that the reported numbers are incorrect.
    5. The perturbation should remove timbre and content while preserving style, i.e. lower T-SIM and raise WER while S-SIM stays high. Table 12 shows the first part (T-SIM 0.85 to 0.17, WER 5.9% to 114%), but S-SIM also falls from 0.91 to 0.51 and DNSMOS from 3.58 to 3.02. By the paper's own metric, the perturbation removes much of the style along with everything else, which contradicts the claim that the reference "still preserves the dominant speaking style".
  3. Important parts of the method are not described well enough. Section 3.2 does not explain the CQT-based pitch representation or the content perturbation setup, and the DPO setup description is quite inadequate with no mention of what TTS systems were used and many other details.

Questions for authors

Here are some clarification questions:

  1. How did each baseline get both a timbre input and a style input? Qwen3-TTS models can either clone a voice (base model) or follow style instructions (voice design or custom voice models) but not both, right? The official StyleTTS2 checkpoints are English-only but half the test sentences are Chinese. What system wrote the style descriptions that were given to CosyVoice3 and Qwen3-TTS in the eval?
  2. Why do the "w/o content" and "w/o timbre" numbers in Table 5 match Table 12 with the labels swapped? Please report the correct numbers.
  3. Where does the timbre reference come from during training?
  4. Are the evaluation clips held out from the training data? What is the source of the clips?
  5. Which emotion recognition model computes S-SIM? The cited Bott et al. (2024) is a TTS paper, and S-SIM is the main style metric, so the exact details are direly needed.
  6. Please provide the missing method details: what the CQT-based pitch representation is, what content perturbation setup are used, which TTS systems produced the DPO samples, etc.

and here are some experiment requests that could make the paper better:

  1. A cascaded baseline i.e. TTS prompted with the style clip, followed by voice conversion to the timbre clip.
  2. Results on a larger test set, with a better S-SIM style metric model, e.g. ParaCLAP or ParaSpeechCLAP.
  3. Evidence that the perturbed references still carry the target style despite S-SIM dropping from 0.91 to 0.51 (Table 12).
10 comments

R2.W1 Contribution 1: Distinguishing dual-reference control from prior work

We agree that independent control of speaker identity and expressive manner has an established research basis, and we do not claim to be the first to propose the dual-reference interface itself. The statement of Contribution 1 in Section 1 of the original manuscript is already qualified:

"We formulate zero-shot TTS with independent style–timbre control as a label-free dual-reference generation problem, where style is modeled as a continuous, reference-driven, and speaker-transferable variable rather than a predefined label, abstract control vector, or prompt-derived description."

Our contribution lies in the label-free formulation of style as a continuous, reference-driven variable, not in introducing the dual-reference interface. Specifically, we focus on constructing style references suitable for dual-reference training without manual style labels or same-style paired speech, enabling reference-driven independent control of timbre and style.

Skerry-Ryan et al. (2018): Also label-free, but learning a fixed-length, absolute-pitch prosody vector from the target speech itself.

We first note that this work shares our motivation. Its §1 explicitly seeks to avoid prosody annotation:

"In order to avoid the challenging problem of schematizing and labeling prosody, we seek methods of modeling prosody that do not require explicit annotations, and present an architecture for learning a latent prosody representation by extracting it from the ground truth speech audio."

Thus, “requiring style labels” is not a distinction between the two methods. The main differences are as follows, based on the descriptions in that paper. (1) The training reference is the target itself, yielding conditional self-reconstruction. Its §3.2 states:

"During training, the reference acoustic signal is simply the target audio sequence being modeled."

No “other utterance” is used as the style reference during training. The desired style information relies entirely on the encoder bottleneck and reconstruction supervision together with the transcript and speaker embedding, without a guarantee of disentanglement; substantial style leakage may also remain in the speaker embedding. (2) Timbre is obtained through an embedding lookup for speakers known during training. Its §3.1 states:

"For each speaker in the dataset, a RdS\mathbb{R}^{d_S} embedding vector is initialized with Glorot (Glorot & Bengio, 2010) initialization."

"we designed a simple speaker identification model that takes varying types of acoustic input, and produces predictions of speaker identity from a universe of speakers known at training time"

Timbre is therefore specified by an embedding table for 44 speakers (§4.1), which substantially limits the feasibility of subsequent large-scale, label-free training. Bian et al. (2019): Motivated by independently controlling multiple styles, with a closed set of categories, same-style pairing, and two additional supervision terms.

This work has a different starting point: it addresses the problem that all styles are compressed into a single representation and cannot be controlled separately. Its §1 states:

"most of the previous researches model all speech styles into one style representation, which contains too much interfering information to be robust and interpretable... can we control different styles independently in an interpretable way?"

Its objective is thus interpretable, independent control of predefined style dimensions, whereas ours is open-set style transfer from arbitrary reference speech. This difference in scope leads to the following three distinctions. (1) Styles form a closed set of categories. Its §2.2 states:

"speaker class is instantiated as 300 different speakers; emotion class is instantiated as happy, sad, angry, fear, confuse, surprise and neutral; prosody class is instantiated as news, story, radio, poetry and call-center."

Style can only represent these predefined instances, and inference-time control interpolates between them (the α interpolation in §2.5). In our method, any speech segment can serve as the style reference. (2) Training relies on same-style pairing. Its §2.3 states:

"we randomly pick two audios with replacement from a group of audios G = {Xi1,∗}, where audios in G share the same style instance i1"

The corpus must contain other recordings with the same style as the target. We construct the reference from the target speech itself and do not depend on such pairing. (3) Disentanglement relies on two additional supervision terms, one requiring class labels. Its §2.4 states:

"our model did not converge well because it was confused by the meanings of multi-reference encoder style embeddings. To address this issue, we introduce a style classification task Lclassification over the reference embeddings."

"we further introduce an orthogonality constraint"

Disentanglement is therefore imposed through a classification loss and an orthogonality constraint, rather than achieved through reference construction itself; the classification loss requires a style-class label for each reference. Our method introduces no additional supervision terms. Appendix A.3 and Table 7 of the original manuscript summarize different control paradigms and do not constitute a claim of priority for the dual-reference interface. We will focus the contribution statement on label-free reference construction and its integration with dual-reference generation training.

R2.W2 Scope of novelty in Contribution 2

Thank you for pointing out the related work. We acknowledge that attribute perturbation has an established research basis. Our contribution is the use of joint timbre, content, and pitch perturbation to construct style conditions for dual-reference TTS. Contribution 2 in Section 1 of the original manuscript states:

"We introduce an attribute perturbation strategy that constructs attribute-purified style references from unpaired training utterances, enabling large-scale label-free style learning without manual style supervision or discrete intermediate style bottlenecks."

The three qualifications—unpaired, without manual style supervision, and without discrete intermediate style bottlenecks—place the novelty claim not on “perturbation” itself, but on using its output directly as an attribute-purified reference for training. The three processed representation streams are resynthesized through SVC into reference speech, while the original speech remains the reconstruction target; the target text and independent timbre reference provide content and identity constraints, respectively.

NANSY (Choi et al., 2021) learns a decomposition of speech attributes through information perturbation, and its original system uses the Yingram pitch representation. We use a CQT-based pitch representation together with cross-speaker timbre replacement, content-feature replacement, and length alignment to construct style references for downstream TTS. The distinction lies in the integration of reference construction with reconstruction supervision, rather than the introduction of a pitch representation alone. Original paper

Gengembre et al. (2024) use an RVC configuration without explicit F0 input to generate converted speech for learning complementary prosody and timbre embeddings. This is a representation and evaluation study and does not involve TTS generation. We explicitly intervene on timbre, pitch, and content, and directly use the resynthesized references for dual-reference TTS training, enabling the model to generate speech from the target text, timbre reference, and style reference. The ideas are related, but the objects and objectives differ. Original paper

Ueda et al. (2024, arXiv:2409.17364) combine voice conversion, timbre perturbation, and metric learning to pretrain a style encoder, and use style-class centroids at inference. We jointly intervene on three attributes and use the processed audio as the style condition; at inference, reference speech can directly specify style without predefined style categories. We will add the citation and clarify these distinctions. Original paper

Conclusion. All three papers are related sources for the components used in our work, and we have cited or will add citations to them. We do not claim perturbation or VC-based data construction itself as a new mechanism. The distinction lies in how these components are combined and in the training objective: label-free reference construction through three-factor perturbation, jointly conditioning a single generation backbone with an independent timbre reference, and an inference interface that accepts arbitrary reference audio.

R2.W3 Contribution 3: The role and scope of BAPC

We agree that BAPC's branch embeddings are conceptually related to BERT's segment embeddings, and that positional offsets do not constitute a new positional encoding principle. Section 3.3 of the original manuscript states:

"This design does not guarantee complete statistical disentanglement by itself; rather, it provides a lightweight structural bias that reduces source ambiguity when multiple acoustic streams are processed by a shared backbone."

The original manuscript thus positions BAPC as a structural bias for reducing acoustic-source ambiguity within a shared backbone, explicitly stating that it does not itself guarantee complete statistical disentanglement. This is distinct from a “dedicated disentanglement mechanism.” Its specific role is also described in the same section:

"BAPC distinguishes the three acoustic sources through branch embeddings and branch-wise positional offsets, namely the timbre reference, target generation state, and style reference."

In Table 5, adding BAPC reduces WER from 2.084% to 1.483% and improves T-SIM from 0.664 to 0.713, while S-SIM decreases from 0.633 to 0.598. This supports its role in improving content accuracy and target-timbre preservation, rather than independently maximizing style similarity. Accordingly, we present BAPC as a conditioning-organization design that supports the overall disentanglement framework, not as a breakthrough in general embedding mechanisms. Nor do we interpret the joint ablation as independent evidence for the respective contributions of branch embeddings and positional offsets.

Scope of the contribution. We will state the main contribution more precisely as constructing multi-attribute-perturbed style references from ordinary speech and learning the respective conditioning roles of text, timbre, and style within a dual-reference generation framework. BAPC is a lightweight design supporting this framework; MeanFlow and DPO serve inference efficiency and preference-based post-training, respectively. We do not base our core novelty claim on “the first use of perturbation,” “the first use of branch embeddings,” or “the first use of dual references.” At the same time, conceptual overlap in basic operations does not replace an assessment of the specific training relationships and experimental outcomes: the ablations in Table 5, together with the same-backbone comparisons in Tables 3 and 4, support the effectiveness of the proposed design for this task, but are not direct performance comparisons with the three papers discussed above.

R2.1 Baseline interfaces, language coverage, and style instructions

Appendix A.3 and Table 7 of the original manuscript describe different style-control paradigms. In the actual experiments, each system uses its corresponding control mechanism; the classification table alone should not be taken to imply that all models natively support the same dual-audio-reference interface.

BaselineControl mechanism used in this work
F5-TTSSingle-reference zero-shot synthesis, used for comparisons of content, naturalness, and efficiency
StyleTTS2Reference-speech-driven style synthesis, evaluated only on the English subset
Vevo2The style reference enters the content-style stage, and the timbre reference enters the acoustic generation stage
IndexTTS2Supports timbre and emotion reference control
CosyVoice3Natural-language instructions describe the style (see Tables 8 and 9 on pages 18–19 for the instructions)
Qwen3-TTSUses Qwen3-TTS-12Hz-1.7B-Base. Reference audio provides timbre conditioning, and manually written text instructions provide style conditioning.

We use the publicly available weights of Qwen3-TTS-12Hz-1.7B-Base. The official generate_voice_clone() interface performs reference-audio-driven timbre cloning. In our experiments, a custom inference script calls the underlying model.generate(), passing the timbre reference, manually written style instruction, and target text in the same inference call. The reference recording specifies the speaker's timbre, while emotion_prompt describes the desired emotion and tone. We will add the model version and this invocation procedure to Appendix A.3.

The StyleTTS2 results correspond only to the English subset, and we will explicitly label its language coverage. The style instructions for both CosyVoice3 and Qwen3-TTS were written manually; their exact content is provided in Appendix B.2 (Tables 8 and 9) of the original manuscript. We will add the instruction source and the actual language coverage of each baseline to Appendix A.3, clearly linking the paradigm overview in Table 7 to the experimental settings.

Reviewer_dWyvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DualRef-TTS, a label-free zero-shot TTS framework that takes separate timbre and style speech references. It constructs training-time style references by perturbing timbre, pitch, and content through an SVC pipeline, uses branch embeddings and positional offsets (BAPC) to distinguish timbre, target, and style streams, and combines MeanFlow generation with DPO-style post-training. Experiments report strong style transfer, timbre preservation, and four-step inference on Chinese/English evaluation grids.

Critical strengths
  1. Independently specifying timbre and open-set style through separate speech references is practically meaningful. The proposed method demonstrates strong subjective style similarity while maintaining competitive timbre similarity and overall speech quality.

  2. The evaluation is broad and targeted, covering controlled swapping, same-backbone comparisons, cross-gender and cross-language settings, and analyses of reference length and temporal order. The temporal analysis provides useful evidence that a concatenated style reference is not simply reduced to a global average.

Critical weaknesses
  1. The central ablation evidence contains an unexplained inconsistency: two crossed perturbation rows in Tables 5 and 12 report nearly identical results despite supposedly evaluating different objects and sample sets. This raises concerns about possible labeling or reporting errors. In addition, the main MeanFlow result includes DPO post-training, so the respective contributions of MeanFlow and DPO cannot be determined from the reported comparisons.

  2. The core learning mechanisms are underspecified. It is unclear how style is preserved when the resynthesis module receives only perturbed timbre, content, and pitch features, especially because some perturbations directly alter cues that the paper itself defines as style. The DPO objective for the MeanFlow generator is also presented without explaining how the required likelihood or surrogate is computed. These omissions limit reproducibility and weaken the support for the claimed gains.

Questions for authors
  1. Why do the crossed rows in Tables 5 and 12 match in all four metrics? Please clarify the evaluated objects and provide corrected tables and per-sample evaluation outputs.
  2. Please report CFM, CFM+DPO, MF, and MF+DPO under otherwise identical settings. Does MF still improve quality/style, or only inference speed, after controlling for DPO?
  3. In Equation (3), through which representation does style enter the SVC resynthesis module? How is preservation of fine-grained pitch and temporal style established after pitch shifting, temporal smoothing, shuffling, and replacement?
4 comments

Response to Reviewer dWyv

R1.1 Corrections to Tables 5 and 12 (Appendix C.1 of the original manuscript) and their evaluation targets

Thank you for identifying this issue. We acknowledge reporting errors in the individual perturbation rows of the original tables. In the revised Table 5, we have removed these three rows while retaining Base, Base w/o BAPC, Base w/o perturbations, and Full (Base + DPO). Table 5 evaluates speech generated by the corresponding dual-reference models within the MeanFlow framework, examining the effects of BAPC, joint attribute perturbation, and DPO. Table 12 (Appendix C.1 of the original manuscript) instead directly evaluates input audio reconstructed under different perturbation configurations, without downstream TTS generation. Its individual perturbation conditions have been corrected to pitch only, content only, and timbre only. The two tables therefore evaluate different objects, and their values should not be interpreted as corresponding results. We apologize for the reporting errors and provide the revised tables below.

Table 5 evaluates the inference outputs of dual-reference models. Each ablation configuration is used to construct style references during training and to train the corresponding dual-reference model. During evaluation, the model generates speech conditioned on the target text, timbre reference, and style reference, and the metrics are computed on the final generated speech. This table examines how each design affects content accuracy, target-timbre preservation, and style transfer. Here, w/o indicates the removal of the corresponding component or perturbation, and all experiments are conducted within the MeanFlow framework. This framework-specific scope is stated in the original caption:

"Table 5: Ablation analysis of key components in the MeanFlow-based framework."

Table 12 (Appendix C.1 of the original manuscript) evaluates input audio reconstructed after perturbation. The object evaluated is stated at the beginning of Appendix C.1 of the original manuscript:

"We randomly sample 1,000 training utterances and evaluate the processed style references themselves, without involving the generation model."

"Table 12 measures residual speaker identity using T-SIM, retained style using S-SIM, residual linguistic content using WER, and signal quality using DNSMOS."

Specifically, it directly measures the retention of timbre, style, and content information in the reconstructed audio relative to the original input, without downstream dual-reference TTS inference. SVC-only involves reconstruction without attribute perturbation; pitch only, content only, and timbre only each enable a single perturbation; Full enables all attribute perturbations. Thus, the two tables evaluate different objects for different purposes, and their values should not correspond one-to-one. The duplicated values and cross-table correspondences in the original manuscript resulted from table-reporting errors.

Table 5: Generation results of ablated models under dual-reference conditioning within the MeanFlow framework.

Model variantWER (%) ↓DNSMOS ↑T-SIM ↑S-SIM ↑
Base1.4833.9110.7130.598
Base w/o BAPC2.0843.8970.6640.633
Base w/o perturbations129.4613.9530.2130.893
Full (Base + DPO)0.9843.9460.7360.624

Table 12: Information retained after SVC reconstruction of the same input audio under different perturbation configurations.

Reference processing conditionT-SIM ↓S-SIM ↑WER (%)DNSMOS ↑
GT——5.23.59
SVC-only (no perturbation)0.8500.9105.93.58
Pitch only0.6550.68510.93.35
Content only0.6500.622120.43.25
Timbre only0.2910.6958.13.53
w/o pitch perturbation0.2240.524115.23.16
w/o content perturbation0.2230.58113.13.35
w/o timbre perturbation0.5040.529172.53.08
Full (all perturbations)0.1740.506114.03.02

In Table 5, a higher T-SIM indicates that the generated speech is closer to the specified target timbre; in Table 12 (Appendix C.1 of the original manuscript), a lower T-SIM indicates weaker cues to the original speaker's identity in the reconstructed reference. WER in Table 12 (Appendix C.1 of the original manuscript) is computed against the original input text; an increase serves to diagnose reduced recoverability of the original content, not improved synthesis quality. In addition, Full in Table 5 denotes the complete model, namely Base with DPO, whereas Full in Table 12 (Appendix C.1 of the original manuscript) only means that all three attribute perturbations are enabled. We will update Section 5.3, Appendix C.1, and the table notes accordingly to make these distinctions explicit.

R1.2 Separating the contributions of CFM, MF, and DPO

Thank you for suggesting that we separate the effects of MeanFlow and DPO. We first clarify the original experimental settings: CFM in the main table did not undergo DPO, whereas MF in the main table is the complete model after DPO training. In the original experiments, DPO was applied only within the MeanFlow framework; following the reviewer's suggestion, we have additionally conducted a CFM+DPO experiment. The role of MeanFlow is stated at the end of Section 3.4 of the original manuscript:

"In our framework, mean-flow is introduced as an efficiency-oriented acoustic generation objective, rather than as the core disentanglement mechanism itself."

Specifically, MeanFlow is intended to reduce the number of inference steps while maintaining comparable synthesis quality and control, rather than to serve as the disentanglement mechanism itself. The CFM and MF rows in Table 1 of the original manuscript correspond to configurations without and with DPO, respectively. Following your suggestion, we have added the CFM+DPO experiment. The results for all four configurations are shown below.

ConfigurationWER (%) ↓DNSMOS ↑T-SIM ↑S-SIM ↑
CFM0.9413.9080.7510.603
CFM+DPO0.9253.9420.7640.619
MF1.4833.9110.7130.598
MF+DPO0.9843.9460.7360.624

DPO improves all four metrics for both CFM and MF. When both use DPO, CFM has a WER lower by 0.059 percentage points and a T-SIM higher by 0.028, while MF has DNSMOS and S-SIM scores higher by 0.004 and 0.005, respectively. In the original manuscript, CFM and MF+DPO use 32 and 4 inference steps, with RTFs of 0.101 and 0.049, respectively. Each has its advantages; the primary role of MeanFlow in this work is to support few-step inference with comparable overall performance (see Appendix E of the original manuscript).

R1.3 The representation pathway through which style enters SVC and evidence for fine-grained style preservation

The style reference in Equation (3) is constructed through joint perturbation of three attributes. The construction of each is described in §3.2 of the original manuscript:

"Timbre perturbation We replace the original speaker-related representation (Hsu et al., 2019) with timbre features sampled from other speakers, thereby explicitly disrupting the shortcut correlation between style and speaker identity."

"Pitch perturbation We adopt a CQT-based pitch representation and apply both global (±12 semitones) and local (±4 semitones) pitch shifts to mitigate residual speaker information, including gender and habitual prosody patterns."

"Content perturbation We perturb the content representation through temporal smoothing, feature shuffling, and cross-sample replacement to prevent the style branch from capturing semantic and local phonetic information."

Specifically, timbre is replaced with features from other speakers, content is smoothed/shuffled/replaced across samples, and CQT-based pitch is shifted globally and segment-wise. The three perturbed representations, f~tim\tilde{f}_{\text{tim}}, f~con\tilde{f}_{\text{con}}, and f~pit\tilde{f}_{\text{pit}}, jointly enter SVC resynthesis through Φ\Phi in Equation (3) of §3.1. Style is not supplied as a separate vector; instead, it is conveyed through the expressive information retained in the three perturbed representations. Regarding “preserving fine-grained pitch and temporal style,” we do not require frame-by-frame reproduction of the source reference’s absolute pitch; instead, bounded global and local pitch shifts are used to weaken speaker-related cues. Content smoothing, shuffling, and cross-sample replacement may affect temporal structure, so we do not claim that all fine-grained temporal information remains unchanged. The extent of style retention should be assessed using the reference-side listening tests and downstream generation results below.

Regarding how style is preserved, we acknowledge that the perturbations do modify some of the cues listed in Appendix B.1. Our claim is that the perturbed references retain sufficient cues for style perception and transfer, not that the organization and absolute values of all cues are fully preserved. Appendix B.1 defines them as follows:

"Different styles may be reflected through acoustic cues such as intonation contour, rhythm, pause pattern, speaking rate, and energy dynamics. These low-level cues are not treated as separate style categories in this paper; instead, they are regarded as observable manifestations that jointly realize a higher-level speaking style."

These cues are thus “observable manifestations” of style, which is jointly realized by their combined pattern. On this basis, the cues listed in B.1 can be compared with the three perturbations as follows:

Cue listed in Appendix B.1Affected by perturbation?Component removedComponent retained
intonation contourYes (pitch perturbation)Absolute pitch is shifted; global translation alone does not remove pitch-range widthRelative direction and variation of within-segment pitch contours; not frame-level fidelity
rhythm, speaking rateMay be indirectly affected by content perturbation and resynthesisNot separately quantifiedComplete preservation is not claimed; useful style retention is assessed through listening tests and generation results
pause patternMay be indirectly affected by content perturbation and resynthesisNot separately quantifiedComplete preservation is not claimed; useful style retention is assessed through listening tests and generation results
energy dynamicsMay be indirectly affected by content perturbation and resynthesisNot separately quantifiedComplete preservation is not claimed; useful style retention is assessed through listening tests and generation results

Pitch requires further clarification: global shifts of ±12 semitones transpose the entire utterance without changing the shape of within-segment contours; local shifts of ±4 semitones are applied segment-wise and introduce bounded steps at segment boundaries. Pitch translation therefore preserves within-segment contour shapes, but local shifts alter relationships between segments; this does not establish complete preservation of the overall contour or frame-level information. For the content perturbation stream, the original manuscript specifies that its targets are semantic and local phonetic information:

"We perturb the content representation through temporal smoothing, feature shuffling, and cross-sample replacement to prevent the style branch from capturing semantic and local phonetic information."

The aim is to reduce the style branch’s reliance on “what is said,” but this does not guarantee that the temporal structure of “how it is said” remains unchanged. In the concatenated-reference experiment in Appendix C.3, the first 40% of the output has an S-SIM of 0.513 with A (versus 0.334 with B), and the last 40% has an S-SIM of 0.485 with B (versus 0.290 with A). This supports the output following the coarse ordering of styles in the reference, rather than establishing complete preservation of fine-grained temporal structure after perturbation. Degradation introduced by SVC resynthesis itself is also negligible:

"GT and SVC-only have nearly identical WER and DNSMOS, showing that SVC resynthesis itself introduces little degradation."

Why absolute pitch is perturbed. The purpose is stated in the perturbation definition in §3.2 of the original manuscript:

"We adopt a CQT-based pitch representation and apply both global (±12 semitones) and local (±4 semitones) pitch shifts to mitigate residual speaker information, including gender and habitual prosody patterns."

Preserving the source reference’s absolute pitch frame by frame may allow the style condition to carry source-speaker pitch cues. We use pitch perturbation to weaken speaker-related associations, including the gender and habitual prosody patterns mentioned in the original manuscript, rather than claiming to remove them completely. Potential conflicts between the source pitch trajectory and the target pitch range or lexical tones are one motivation for this design; we have not independently established that such conflicts necessarily cause tone distortion in Chinese. The cost is measurable, and we report it transparently. S-SIM in Table 12 drops from 0.910 for SVC-only to 0.506 for Full, and SS-SIM in the reference-side listening test drops from 4.86 to 3.78. These changes reflect the aggregate cost of full attribute perturbation; they cannot be attributed specifically to absolute-pitch changes, nor do they establish that only speaker-related information is lost. Accordingly, our conclusion is limited to the perturbed references still retaining perceptible, transferable style, rather than lossless preservation of every style cue or frame-level fidelity. To directly examine the perturbed references, we have added a subjective evaluation of the reference audio itself: using the original references for comparison, the following two processed versions are rated under the same evaluation guidelines. We apply two processing conditions to the same set of original input audio: one undergoes SVC resynthesis without attribute perturbation; the other undergoes full attribute perturbation followed by resynthesis through the same SVC pipeline.

Style-reference processingNaturalness S-MOS ↑Timbre similarity ST-SIM ↑Style similarity SS-SIM ↑
SVC processing only (no attribute perturbation)3.98 ± 0.054.72 ± 0.034.86 ± 0.03
Full attribute perturbation2.28 ± 0.061.82 ± 0.093.78 ± 0.08

After full perturbation, timbre similarity decreases from 4.72 to 1.82, while style similarity remains at 3.78, indicating that the intended style remains perceptible when the original speaker's timbre is substantially suppressed. Content perturbation introduces some distortion into the reconstructed audio, which may also affect listeners' judgments of style and lower subjective style scores. The naturalness decrease from 3.98 to 2.28 is consistent with signal degradation, but this aggregate comparison cannot attribute the entire decrease in style scores to distortion. We therefore interpret the results as retention of useful style information, rather than lossless preservation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.