acceptodds
Under review as a conference paper at ICLR 2027

CookVoice: An Unified Framework for Multi-Modal Style Controllable Human Voice Generation

Abstract

Human voice generation has made rapid progress in speech generation, singing voice generation, voice cloning, and voice editing. However, most existing systems are designed for specific tasks and often rely on task-dependent architectures, control signals, or autoregressive decoding, limiting fine-grained controllability and inference efficiency. In this paper, we propose CookVoice, a unified framework for multimodal, multi-style, and multi-task human voice generation. CookVoice decomposes the human voice into three key factors: content, prosody, and style, enabling both speech and singing voice generation within a unified model. To achieve precise and flexible controllability, we design a flexible alignment strategy that maps text, style, and prosody control signals onto the frame-level of spectrogram. This design allows CookVoice to support a wide range of tasks, including text-to-speech, text-to-singing voice, style-controllable generation, voice mimicry, voice conversion, and voice editing. Experimental results show that CookVoice achieves generation quality comparable to existing Text-to-Speech and text-to-singing voice baselines, while providing stronger style and prosody controllability. Moreover, CookVoice achieves competitive performance to large-scale baselines with only million parameters and efficient inference using as few as 4 ODE steps, making it a practical solution for real-world human voice generation applications.

Primary area:
generative models
Keywords:
voice generation, generative artificial intelligence, speech generation, singing voice generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KLgDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CookVoice, a unified non-autoregressive framework for controllable speech and singing voice generation. The model conditions generation on linguistic content, style, and prosody. Style can be specified through text descriptions or reference audio, while prosody can be provided through discrete linguistic/musical tokens or continuous F0 contours. These conditions are expanded to the acoustic frame level and concatenated to condition a flow-matching Diffusion Transformer operating in a learned acoustic latent space. Random switching between conditioning sources enables multiple control configurations within one model. Experiments on bilingual speech and singing datasets report strong style-similarity and pitch-following results with a compact model, alongside trade-offs in perceptual quality and intelligibility.

Critical strengths
  1. A practical and flexible conditioning interface. CookVoice combines speech and singing generation with text-/reference-based style conditioning and discrete/continuous prosody control within a shared architecture. This flexibility is useful for applications requiring different combinations of content, style, and pitch control.

  2. Promising controllability with a compact model. The reported 43.51M-parameter model achieves strong style-similarity and pitch-following results across several conditioning configurations. The small number of sampling steps also makes the approach potentially attractive where computational resources are limited. These results demonstrate practical merit, even though they do not by themselves establish methodological novelty.

Critical weaknesses
  1. The methodological novelty of the unified formulation and alignment mechanism is insufficiently established. Joint speech and singing generation is already supported by prior systems, including the Vevo2 baseline discussed in this paper. The additional combination of conditioning interfaces is useful, but the paper does not clearly identify the conceptual or technical advance beyond this integration. In particular, the “flexible frame-level alignment” in Sections 3.2–3.4 appears to consist of duration-based expansion of phoneme and note embeddings, repetition of a global style embedding, and concatenation of the resulting frame-level conditions. Speech alignment and duration prediction rely on ParaStyleTTS, while singing durations are derived from score timing. It remains unclear what new alignment mechanism is introduced and which limitations of conventional duration-based conditioning it specifically overcomes. This is central to my assessment because the proposed alignment is presented as a main technical contribution.

  2. The experimental evidence does not isolate or sufficiently substantiate the central contribution. The conditioning-source comparisons and ODE-step analysis characterize the complete system, but they do not isolate the benefit of the proposed alignment design. A comparison against conventional alignment or conditioning alternatives using the same backbone, training data, and input information is missing. Likewise, the experiments do not establish whether joint speech–singing training provides benefits over task-specific training. The reported controllability gains should also be distinguished from overall generation quality: in Table 6, CookVoice’s voice-plus-continuous-prosody configuration has a TTS MOS of 3.98 versus 4.42 for IndexTTS, and singing WERs of 10.01/24.37 for Chinese/English versus 4.66/11.37 for Vevo2. These trade-offs are not disqualifying by themselves, but they make stronger evidence for the claimed methodological contribution particularly important. Evaluation details—including speaker-disjoint splits, reference–target pairing, and subjective-test sample sizes—also need clarification to establish the scope and reliability of the results.

Questions for authors
  1. What specific operation distinguishes the proposed flexible alignment from conventional duration-based expansion and frame-level conditioning? Can you identify a concrete case that the proposed mechanism handles but the conventional alternative cannot?

  2. Is there controlled experimental evidence isolating the contribution of this alignment mechanism, with the backbone, training data, and conditioning information held fixed? Is there evidence that joint speech–singing training improves upon task-specific training?

1 comment

Dear reviewer KLgD. Thank you for your constructive feedback. We have carefully answered each of your questions point by point below.

Q1: Methodological novelty of the unified formulation and alignment mechanism.

Recent research has increasingly moved toward autoregressive approaches with large AR models and large-scale training to learn temporal structures and dependencies through token prediction. While this direction has shown impressive progress, CookVoice explores a different approach. Rather than relying on the model to learn everything implicitly from large amounts of data, we explicitly incorporate available alignment, content, and control information into a lightweight generative framework.

Our motivation is to explore how a relatively simple model can go with careful conditioning design and limited training data, while still supporting diverse capabilities such as TTS, TTSV, multimodal style control, and fine-grained prosody control. Duration-based expansion is one example of this design philosophy, providing a simple way to temporally align different conditioning signals.

We believe CookVoice shows that, even without large-scale training, a lightweight model trained on a relatively small amount of well-annotated data can still support a competitive range of capabilities. Rather than competing with large-scale approaches through model scaling, our work highlights the potential of making better use of the information already available.

Q2: Scaling and Further Direction. Scaling CookVoice is part of our ongoing work, but the availability of well-annotated and aligned training data remains a major bottleneck. This is also an advantage of autoregressive approaches, which can learn temporal dependencies more implicitly from large-scale data without requiring explicit alignment annotations.

One promising direction is to incorporate a recognition model that can automatically extract and temporally align content and prosody information from raw audio, reducing the reliance on annotated data and enabling further scaling. However, we are still exploring suitable models and approaches to achieve this reliably.

Reviewer_zN4iOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds a unified speech and singing generation model named CookVoice, by three frame-aligned conditions on a DiT model. Specifically, an embedding from text prompt or from a reference voice provides global style control, an upsampled phoneme embedding for speech content, and a prosody embedding for detailed prosody control (discrete tokens for MIDI notes or continuous F0 contour). The style source and prosody source are each randomly switched during training to make the model conduct TTS, singing synthesis and other tasks.

Critical strengths

A lightweight DiT model for controllable speech generation.

Critical weaknesses

Below are the main shortcomings: a) Limited technical novelty: compared with similar approaches (multi-conditional unified speech generation) in earlier TTS models, the main difference is the DiT backbone. b) Many claimed tasks lack a quantitative evaluation or a comparison with other systems, like voice editing and VC. c) Inconsistencies and errors in reported numbers. As an example, sec. 5.1 says Vevo2 has 0.25 F0-Corr for TTS, but the value is 0.12 in Table 5. d) Missing details of the model, like how to obtain F0 conditions during inference for TTS?

Questions for authors

Q1: How are the generated and reference F0 sequences aligned when their lengths differ (e.g., for baselines with their own duration predictors)? It seems the ground-truth duration is used during inference. Q2: At inference, does CookVoice use predicted durations or ground-truth/aligner durations? For TTSV, where does the total frame count come from? Q3: Can you quantitatively evaluate at least VC/SVC and speech editing against dedicated baselines? Q4: There are a lot open-source dataset, why does the experiment use such a small dataset?

1 comment

Dear reviewer zN4i. Thank you for your constructive feedback. We have carefully answered each of your questions point by point below.

Q1: Many claimed tasks lack quantitative evaluation, such as voice editing and VC.

We have added additional quantitative evaluations for voice conversion (VC) and editing. Please check common response for more details.

Q2: Inconsistencies in reported numbers.

Thanks for point out, the discrepancy in Vevo2's F0-Corr between Section 5.1 and Table 5 is a typo and we will correct in the revised manuscript.

Q3: How are F0 conditions obtained during TTS inference?

F0 conditioning is optional and is particularly useful for tasks such as prosody transfer, where we aim to generate the same linguistic content with a different prosodic pattern. In such cases, F0 can be extracted from a reference voice using pitch estimation models such as CREPE and incorporated into CookVoice as a continuous prosody control signal. When the reference and target have different lengths, the extracted F0 contour can be resized through linear interpolation to match the target voice length.

Q4: How are generated and reference F0 sequences aligned when their lengths differ?

We use linear interpolation to resample the F0 sequences to the same temporal length before computing F0-Corr. This accommodates differences in predicted durations while preserving the overall pitch variation patterns.

Q5: Does CookVoice use predicted or ground-truth durations during inference? How is the total frame count determined for TTSV?

CookVoice supports both predicted and alignment-derived durations depending on the task: standard TTS uses predicted durations, while precise speech editing can use reference-aligned durations to preserve the original timing outside edited regions. For TTSV, the total frame count can from reference audio or estimated from the musical score using its tempo (BPM) and note durations. This flexibility allows users to choose between generall TTS generation and temporally controlled editing.

Q3: More Ablations

We have added additional quantitative evaluations for voice conversion and voice editing, including comparisons with dedicated baselines.

Q4: Dataset scale

While large-scale speech datasets are widely available for speech, singing voice datasets with annotated musical scores and accurate note timings remain limited. Given that CookVoice is not solely focused on speech-related tasks but also supports singing voice generation, we selected this dataset to accommodate both tasks.

Reviewer_vXLGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed a CookVoice model, which is a unified non-autoregressive framework for multi-task human voice generation. This model has three factors: content, prosody, and style; the authors apply a frame-level alignment strategy. Experimental results demonstrate that the proposed method is better than several baseline models.

Critical strengths
  1. CookVoice unifies TTS, TTSV, style-controllable generation, and voice conversion in one small model.
  2. CookVoice has remarkable efficiency: with 4.99% of Vevo2's parameters, it can achieve strong CookVoice controllability.
  3. The framework of the proposed model is clearly represented.
Critical weaknesses
  1. The contributions of this paper are limited. The core of the proposed "frame-level alignment" is an engineering integration rather than a fundamentally new approach, especially given that Vevo2 already unified TTS and TTSV.
  2. The authors need to explain the experiments more clearly; WER is higher than multiple baselines, and the paper does not explain why Vevo2 achieves a lower WER under the same evaluation.
  3. Some ablation studies are missing. The authors need to add more ablation studies that show the performance degradation if frame-level alignment is removed or replaced with simpler global conditions, and it is unknown whether the claimed contribution is really necessary.
  4. Some comparisons are not fair. For example, S-SIM uses the authors' own style encoder (AutoSIFT), and MOS's evaluation protocol is undisclosed. CookVoice is trained on only 168 hours of data, while baselines like Vevo2 and IndexTTS use substantially larger datasets, making the MOS gap potentially attributable to data scale rather than method design.
  5. The authors need to provide some failure cases and explain the reasons. Also, the authors need to discuss the boundary conditions.
  6. The paper claims support for 13+ tasks (Table 3), but experiments only evaluate TTS and TTSV—tasks like voice conversion, speech editing, humming-to-voice, and sketch-to-voice are never tested.
  7. It is necessary to perform an analysis of different backbones on the results, not just one DiT-S.
Questions for authors

Please see weaknesses; especially, I want to know how the authors can address my comments.

2 comments

Dear reviewer vXLG. Thank you for your constructive feedback. We have carefully answered each of your questions point by point below.

Q1: Difference with Vevo2.

Unlike Vevo2, CookVoice is a None-Autoregressive model with smaller size that offer comparable generative performance and offer higher flexible controllability. Vevo2 relies on voice only for style prompting, CookVoice supports multimodel style control through both text and voice madality. Our contribution lies in the flexible and controllable architectural design for voice generation.

Q2: Higher WER compared with baselines.

CookVoice is trained on only small data-scale. As acknowledged in our limitations, CookVoice has not yet been scaled up, the WER gap can come from this.

Q3: Missing alignment ablations.

We provide ablation results in the appendix to investigate how different conditioning signals (discrete or continuous tokens for prosody, different source of style signal) affect performance (Figure 6 and Tables 6 and 7). These results help clarify the contribution of our frame-level conditioning design.

Regarding comparisons with other alignment approaches, such as unaligned conditioning and filler-based alignment used in E2-TTS and F5-TTS, we include F5-TTS as a representative baseline for comparison.

Q4: Fairness of comparisons and evaluation protocols.

As the review process is double-blind, we believe any discussions of author identity should be avoided during rebuttal period. Regarding S-SIM, AutoSIFT was selected because it extracts disentangled speaking-style representations, whereas conventional speaker verification models is trained on to capture speaker identity, which does not adequately reflect the style controllability investigated in CookVoice.

Q5: Failure cases and boundary conditions.

The primary limitation of CookVoice is its relatively small training scale, which can affect intelligibility and generalization. Discussion about limitation and failure scope is presented in Appendix G.

Q6: Limited evaluation of the claimed tasks.

The primary limitation of CookVoice is its relatively small training scale, which may affect intelligibility, as reflected in the Word Error Rate (WER). Another limitation is the unintended transfer of background artifacts alongside the intended speaking style. We provide further discussion of these limitations and failure cases in Appendix G and Common Response.

Q7: Evaluation with different backbones.

Our current experiments focus on DiT-S to evaluate the effectiveness of the proposed framework under limited computational resources. We acknowledge that evaluating larger models and alternative backbones would provide further insights into the generalizability of our approach. We consider this an important direction for future work, as also discussed in the limitations section.

Thanks for the rebuttal, but my concerns remain. I will maintain my original rate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.