- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CookVoice, a unified non-autoregressive framework for controllable speech and singing voice generation. The model conditions generation on linguistic content, style, and prosody. Style can be specified through text descriptions or reference audio, while prosody can be provided through discrete linguistic/musical tokens or continuous F0 contours. These conditions are expanded to the acoustic frame level and concatenated to condition a flow-matching Diffusion Transformer operating in a learned acoustic latent space. Random switching between conditioning sources enables multiple control configurations within one model. Experiments on bilingual speech and singing datasets report strong style-similarity and pitch-following results with a compact model, alongside trade-offs in perceptual quality and intelligibility.
Critical strengths
-
A practical and flexible conditioning interface. CookVoice combines speech and singing generation with text-/reference-based style conditioning and discrete/continuous prosody control within a shared architecture. This flexibility is useful for applications requiring different combinations of content, style, and pitch control.
-
Promising controllability with a compact model. The reported 43.51M-parameter model achieves strong style-similarity and pitch-following results across several conditioning configurations. The small number of sampling steps also makes the approach potentially attractive where computational resources are limited. These results demonstrate practical merit, even though they do not by themselves establish methodological novelty.
Critical weaknesses
-
The methodological novelty of the unified formulation and alignment mechanism is insufficiently established. Joint speech and singing generation is already supported by prior systems, including the Vevo2 baseline discussed in this paper. The additional combination of conditioning interfaces is useful, but the paper does not clearly identify the conceptual or technical advance beyond this integration. In particular, the “flexible frame-level alignment” in Sections 3.2–3.4 appears to consist of duration-based expansion of phoneme and note embeddings, repetition of a global style embedding, and concatenation of the resulting frame-level conditions. Speech alignment and duration prediction rely on ParaStyleTTS, while singing durations are derived from score timing. It remains unclear what new alignment mechanism is introduced and which limitations of conventional duration-based conditioning it specifically overcomes. This is central to my assessment because the proposed alignment is presented as a main technical contribution.
-
The experimental evidence does not isolate or sufficiently substantiate the central contribution. The conditioning-source comparisons and ODE-step analysis characterize the complete system, but they do not isolate the benefit of the proposed alignment design. A comparison against conventional alignment or conditioning alternatives using the same backbone, training data, and input information is missing. Likewise, the experiments do not establish whether joint speech–singing training provides benefits over task-specific training. The reported controllability gains should also be distinguished from overall generation quality: in Table 6, CookVoice’s voice-plus-continuous-prosody configuration has a TTS MOS of 3.98 versus 4.42 for IndexTTS, and singing WERs of 10.01/24.37 for Chinese/English versus 4.66/11.37 for Vevo2. These trade-offs are not disqualifying by themselves, but they make stronger evidence for the claimed methodological contribution particularly important. Evaluation details—including speaker-disjoint splits, reference–target pairing, and subjective-test sample sizes—also need clarification to establish the scope and reliability of the results.
Questions for authors
-
What specific operation distinguishes the proposed flexible alignment from conventional duration-based expansion and frame-level conditioning? Can you identify a concrete case that the proposed mechanism handles but the conventional alternative cannot?
-
Is there controlled experimental evidence isolating the contribution of this alignment mechanism, with the backbone, training data, and conditioning information held fixed? Is there evidence that joint speech–singing training improves upon task-specific training?
Dear reviewer KLgD. Thank you for your constructive feedback. We have carefully answered each of your questions point by point below.
Q1: Methodological novelty of the unified formulation and alignment mechanism.
Recent research has increasingly moved toward autoregressive approaches with large AR models and large-scale training to learn temporal structures and dependencies through token prediction. While this direction has shown impressive progress, CookVoice explores a different approach. Rather than relying on the model to learn everything implicitly from large amounts of data, we explicitly incorporate available alignment, content, and control information into a lightweight generative framework.
Our motivation is to explore how a relatively simple model can go with careful conditioning design and limited training data, while still supporting diverse capabilities such as TTS, TTSV, multimodal style control, and fine-grained prosody control. Duration-based expansion is one example of this design philosophy, providing a simple way to temporally align different conditioning signals.
We believe CookVoice shows that, even without large-scale training, a lightweight model trained on a relatively small amount of well-annotated data can still support a competitive range of capabilities. Rather than competing with large-scale approaches through model scaling, our work highlights the potential of making better use of the information already available.
Q2: Scaling and Further Direction. Scaling CookVoice is part of our ongoing work, but the availability of well-annotated and aligned training data remains a major bottleneck. This is also an advantage of autoregressive approaches, which can learn temporal dependencies more implicitly from large-scale data without requiring explicit alignment annotations.
One promising direction is to incorporate a recognition model that can automatically extract and temporally align content and prosody information from raw audio, reducing the reliance on annotated data and enabling further scaling. However, we are still exploring suitable models and approaches to achieve this reliably.