DualRef-TTS: A Label-Free Dual-Reference Framework for Independent Style-Timbre Control in Zero-Shot Speech Synthesis

Zero-shot text-to-speech (TTS) systems have achieved significant progress in both speaker similarity and speech naturalness. Building on these advances, recent work has further explored more independent control over emotion and speaking style through a variety of paradigms. However, most existing approaches still rely on human-defined style taxonomies, which limits their ability to achieve reliable dual-reference control, scalable label-free learning, and robust style control for in-the-wild scenarios. We introduce DualRef-TTS, a label-free dual-reference paradigm for independent style–timbre control in zero-shot speech synthesis, where style is learned as a continuous, reference-driven, and speaker-transferable variable, rather than as a predefined label, abstract control vector, or prompt-derived description. First, attribute perturbation constructs attribute-purified style references for label-free style supervision. Second, Branch-Aware Positional Conditioning (BAPC) combines branch embeddings with branch-wise positional offsets to distinguish the timbre reference and style reference, thereby reducing cross-branch interference. Building on this architecture, we further adopt a mean-flow-based acoustic generation method to reduce inference steps while preserving synthesis quality. Finally, Direct Preference Optimization (DPO)-style post-training further aligns the generation distribution with style alignment, semantic accuracy, and prosodic naturalness. Experimental results demonstrate that the proposed method is comparable to state-of-the-art (SOTA) systems across standard metrics, while introducing a label-free dual-reference paradigm that avoids intermediate representations and prompt-based control. This supports more flexible and independent control of timbre and style in open-domain speech synthesis. Audio samples are available at: https://dualref-tts.github.io/