acceptodds
Under review as a conference paper at ICLR 2027

Pushing Frontier of Lightweight Diffusion for Zero-Shot Text-to-Speech

Abstract

Zero-shot text-to-speech (TTS), which clones an unseen voice from a short prompt, has largely relied on domain-specific priors such as forced alignment or phoneme duration, whose supervision is costly and hard to scale. Recently, large diffusion models trade these priors for scale, letting the alignment emerge from data, while lightweight models for on-device deployment cannot afford this trade. In the absence of a prior, such models fail to learn the alignment reliably, and the explicit priors would reimpose the supervision that scaling was meant to eliminate. In this paper, we present **LiteMel**, a family of mel-spectrogram diffusion models from 0.03B (Small) to 0.13B (Large) parameters that achieves a Pareto-optimal trade-off between model size and performance. Specifically, we first propose a dual-path framework that fuses plain Diffusion Transformers (DiT) favouring intelligibility with hierarchical DiTs favouring speaker fidelity. We also introduce speaker identity that enters through the time embedding and reaches every block by adaptive normalization. Moreover, we propose a learned soft text-speech aligner that restores the monotonic guidance without any supervision. It contains a duration head with a differentiable Gaussian expansion trained by the flow-matching objective alone. To further enhance, we introduce on-policy distillation (OPD) for each student model against our 0.13B member as teacher along the student's own stochastic rollouts, so that the teacher corrects the states the sampler actually visits. Our extensive experiments show that LiteMel-Large attains the SOTA intelligibility, and OPD improves in every metric at both student sizes, cutting the average error by 23% relative to pre-training. The three models together occupy the entire Pareto frontier of average error against parameter count.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-to-Speech, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sstKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces LiteMel, a family of lightweight zero-shot text-to-speech models ranging from 0.03B to 0.13B parameters. It combines a dual-path Diffusion Transformer, speaker-aware adaptive normalization, and soft monotonic alignment learned without external duration labels. The authors also apply on-policy distillation (OPD), using student-generated trajectories to transfer knowledge from a larger teacher. Trained on approximately 100K hours of English and Chinese speech, LiteMel is evaluated on Seed-TTS benchmarks, subjective listening tests, and efficiency measurements. LiteMel-Large reports 2.87% average intelligibility error, while OPD reduces average error by 11.7% and 23.6% for the Base and Small models, respectively. The paper aims to achieve competitive speech quality with substantially fewer parameters.

Critical strengths
  1. On-policy distillation shows useful within-model improvements. Both distillation arms at each size start from the same supervised checkpoint, which makes the reported reductions in average error at 0.03B and 0.06B, with simultaneous improvements in speaker similarity, meaningful within each model size and independent of the baseline comparisons. These results suggest that student-trajectory post-training can benefit compact TTS models, although the rollout implementation requires the clarification requested below.

  2. Clear improvements from alignment and speaker conditioning. Table 7 shows that adding soft monotonic alignment reduces average error from 5.03% to 3.37%, with a 4.44-point reduction on test-hard, while speaker-aware AdaLN further improves both average error and speaker similarity. These ablations provide useful evidence for the individual components, even though they do not establish the necessity of the complete dual-path architecture.

Critical weaknesses
  1. Unresolved mathematical formulation and inference procedure. Section 3.3 defines a decreasing sampling grid with Δtk=tk+1−tk<0\Delta t_k = t_{k+1} - t_k < 0, but Equation (5) uses this negative step as the Gaussian covariance g2ΔtkIg^2 \Delta t_k I. Equation (7) consequently produces a negative KL coefficient c(tk)2Δtk/2g2c(t_k)^2 \Delta t_k / 2g^2, which no divergence can take. The unweighted loss in Equation (6) also differs from the weighted objective the derivation produces, and the paper does not state whether w(tk)w(t_k) is applied. More importantly, Equation (2) requires the target mel-spectrogram length NmelN_{mel}, and the paper states there is no separate length prediction, yet it does not explain how this length is determined at inference for unseen text. If reference-derived lengths were used during evaluation, the reported performance would depend on information unavailable in real-world synthesis. These unresolved issues leave an essential part of the synthesis and evaluation pipeline unverifiable.

  2. Insufficient architectural validation and unclear experimental comparisons. Table 7 shows that adding the dual-path backbone increases average error from 4.47% to 5.03% and reduces UTMOS from 3.25 to 2.98, although speaker similarity improves from 0.693 to 0.704. The incremental ablation therefore does not establish whether the dual-path architecture is necessary or beneficial in the final model, and this trade-off is not acknowledged in the text. The proposed Gaussian alignment also resembles the Gaussian upsampling and unsupervised duration modeling of Non-Attentive Tacotron (Shen et al., 2020), while the OPD formulation builds on existing diffusion distillation work that the paper itself cites. The paper does not establish what is methodologically new beyond their application and combination. Furthermore, Tables 1 and 4 report different test-hard CER values for the smaller models and for F5-TTS without explaining the evaluation configurations, which leaves the provenance of the Table 1 baselines unclear. The small average-error advantage over dots.tts (2.87% versus 2.92%) is reported without uncertainty analysis, and the +1.14 CMOS result over ground-truth speech requires clarification of the listening protocol. Missing parameter-counting and runtime documentation further weakens reproducibility.

Questions for authors
  1. Equations (5) and (7) appear to use a negative time step for Gaussian covariance and KL weighting. Could the authors clarify the sampling direction, correct the formulation, and explain the weighting in Equation (6)?

  2. How is the target mel-spectrogram length in Equation (2) determined during inference, and were reference-derived lengths used during evaluation?

  3. Why do Tables 1 and 4 report different test-hard CER values for Small (6.47% vs. 7.12%), Base (6.19% vs. 6.33%), and F5-TTS (7.44% vs. 7.56%)? Were different checkpoints or evaluation settings used?

  4. Given the adverse error and UTMOS results in Table 7, what is the specific contribution of the dual-path backbone? How do the alignment and OPD methods differ from prior work, and is the improvement over dots.tts statistically reliable?

  5. How should the +1.14 CMOS result for Base+OPD over ground-truth speech be interpreted, and what evaluation protocol and statistical evidence support it?

Reviewer_wvcUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper discusses LiteMel, a family of small models for generating speech using an audio prompt. LiteMel uses two processing paths and aligns text and speech without external duration labels. It also uses knowledge distillation from a larger model to help smaller models perform better. Tests on English and Chinese speech show that LiteMel matches the word accuracy of many other systems but uses far fewer parameters.

Critical strengths
  1. LiteMel is able to produce clear speech even with few parameters. This makes it useful for speech generation when model size and computing power are limited.
  2. Further, it does not require domain-specific priors such as forced alignment. The alignment mechanism learns from the speech generation itself and doesn't require extra duration labels.
Critical weaknesses
  1. In Equation 5, $\Delta t_{k} < 0 $ with a negative time increment; this becomes a negative variance. Similarly, the same negative increment makes the stated KL divergence negative.
  2. In ablation studies, Table 3 shows that adding the dual-path backbone increases Avg.Err but slightly improves speaker similarity. It also appears that the main improvement comes from adding alignment. Therefore, to prove that the complex backbone really helps, it is required to compare the proposed method with plain DiT under the same alignment and speaker conditioning.
  3. The demo samples I inspected in the supplementary material contain only very short phrases. What generation lengths does the method target, and what is the duration distribution of evaluation utterances? Could you please provide representative full-sentence and longer samples so we can assess prosody, speaker consistency, and alignment stability beyond short phrases?
  4. During distillation, the model adds random noise at each generation step, but not during evaluation. I think the authors should clarify the claim that the two processes 'strictly match.'
  5. Since several confidence intervals in Table 2 include zero, the listening results do not clearly show a preference in those comparisons.
Questions for authors
  1. It would be helpful to clarify how the total output duration is determined at Inference ?
  2. Post-training is outlined in Figure 3; the text discusses Figure 5.
  3. Missing reference to Appendix (??) in the reproducibility statement.
Reviewer_hSM4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates lightweight zero-shot text-to-speech and proposes LiteMel, a family of flow-matching models ranging from 0.03B to 0.13B parameters. The framework combines a dual-path DiT backbone, a learned soft text–speech aligner without external duration supervision, and speaker-conditioned adaptive normalization. Experiments demonstrate competitive performance with substantially fewer parameters.

Critical strengths

The paper addresses an important practical problem: reducing the size of zero-shot TTS models while maintaining speech quality.

The proposed alignment module learns the correspondence between text and speech without additional duration labels or external alignment tools.

The experiments cover different model sizes and include both automatic metrics and human evaluations.

Critical weaknesses

The paper lacks comparisons with other 0.03B and 0.06B models trained under the same conditions. It is therefore unclear whether the improvements come from the proposed design or could also be achieved by simpler models with the same training and distillation.

The benefit of the dual-path architecture is not fully demonstrated. In Table 3, adding this component increases the average error rate. A comparison with a plain DiT using the same alignment and speaker-conditioning modules would help clarify its contribution.

The paper combines several established techniques, but the distinctive contribution of their combination remains unclear.

Questions for authors

The time-step convention in Eq. (5) appears inconsistent: a negative time increment is used in the Gaussian covariance. Please clarify the equation and whether the same issue affects the implementation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.