- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper builds a streaming TTS model that generates blocks of continuous speech latents with one to three network evaluations per block. It adapts MeanFlow training to include text guidance and adds a short critic fine-tuning stage to improve one-step output. On LibriSpeech-PC, the model shows low word error rates, an 18 ms time to first audio, and higher throughput than the tested systems. Speaker similarity remains well below the stronger baselines. (Sections 3–8; Table 5.)
Critical strengths
- The blockwise design gives a clear reduction in startup delay and improves throughput as the batch grows.
- The guidance and critic ablations are useful, including a control that gets the same extra training without the critic.
Critical weaknesses
-
MeanFlow and patchwise continuous speech generation already exist. The guided target is useful, but Table 5 omits DiTAR and CALM/Pocket TTS, which Section 2 identifies as close designs. The matched flow-matching control establishes the benefit of fewer steps, but does not establish an advantage over comparable few-step streaming systems. This leaves the size of the contribution uncertain.
-
Evaluation is limited to LibriSpeech-PC test-clean, and no aggregate listener ratings are reported. Table 5 gives speaker similarity of 0.366, versus 0.554 for F5-TTS after the same codec round trip. Both the critic and the similarity metric also rely on WavLM. The automatic quality scores therefore leave open whether the speed gain preserves satisfactory voice identity and naturalness.
Questions for authors
Were DiTAR or CALM/Pocket TTS evaluated under the same timing and quality protocol? If results are available, where does this model fall on their speed-quality trade-off?
Does the listening bundle have collected listener ratings, or is there an independent speaker-similarity evaluation that supports the critic’s gains beyond the WavLM-based metric?
5 comments
Re: Comparison with CALM / Pocket TTS and DiTAR (Q1)
We thank the reviewer for this question. In short: on matched data our model matches the Pocket TTS recipe on the speaker-similarity metric at 1.7 times its throughput (block 4, two steps), or runs 3 to 4.4 times faster at somewhat lower similarity (block 8, one step).
How the two designs relate. Pocket TTS works in the same latent space as our model (Mimi) but generates one 80 ms frame per step, where ours generates a block of eight frames (640 ms). In the terms of our paper it has a block size of 1. Our block-size analysis (Fig. 5, block sizes 4 to 32) shows that shorter blocks are more accurate in one step and slower, because each second of audio takes more network evaluations; Pocket TTS sits past the short end of that range. Our paper addresses the opposite end: how to generate a long block in a single evaluation, which is where the throughput comes from and what the guided target and the critic are for.
What we compared. The released Pocket TTS checkpoint is a 6-layer student distilled from a 24-layer model trained on 88k h, and the 24-layer model is not released. To compare on matched data, we used Kyutai's training code to retrain their recipe (the 24-layer model, their sampler defaults) on LibriSpeech 960 h. Speed is measured on one B300 under the paper's timing protocol (Appendix D).
| matched data: LibriSpeech 960 h | audio per step | WER % | SIM | PQ | audio-s / s, batch 1 | batch 16 |
|---|---|---|---|---|---|---|
| Pocket TTS recipe (24 layers), retrained | 80 ms | 2.15 | 0.439 | 6.63 | 39 | 352 |
| Ours, block 4, 2 steps | 320 ms | 2.76 | 0.443 | 6.42 | 67 | 601 |
| Ours, block 8, 1 step | 640 ms | 2.91 | 0.396 | 6.58 | 172 | 1058 |
| Ours, block 8, 2 steps | 640 ms | 2.88 | 0.420 | 6.42 | 125 | 888 |
| Ours, block 16, 1 step | 1.28 s | 4.23 | 0.357 | 6.49 | 304 | 1416 |
| larger data | audio per step | WER % | SIM | PQ | audio-s / s, batch 1 | batch 16 |
|---|---|---|---|---|---|---|
| Pocket TTS, released (distilled 6-layer student; 88k h) | 80 ms | 1.55 | 0.493 | 6.75 | 133 | 1036 |
| Ours trained on Emilia (137k h, only ~2 epochs), block 8, 1 step | 640 ms | 2.19 | 0.455 | 6.40 | 172 | 1058 |
| Ours trained on Emilia (137k h, only ~2 epochs), block 8, 2 steps | 640 ms | 2.00 | 0.487 | 6.21 | 125 | 888 |
We observe the following.
- Matched data. Block 4 with two steps reaches a similarity of 0.443, against 0.439 for the Pocket TTS recipe, while generating four times more audio per step and running 1.7 times faster. The Pocket TTS recipe keeps a lower WER (2.15 % against 2.76 %).
- Block size is a dial. Longer blocks trade similarity for speed. At one step, similarity is 0.420, 0.396 and 0.357 for blocks 4, 8 and 16, and block 8 is 3 to 4.4 times faster than the Pocket TTS recipe. Pocket TTS, at one frame per step, is consistent with this trend.
- Larger data. Trained on Emilia, our two-step model reaches 0.487, against 0.493 for the released Pocket TTS. That model gets its speed from distillation into a network a quarter of the size (90M parameters; ours has about 330M, the size of their 24-layer model). Ours reaches comparable speed without distillation, because it generates 0.64 s per evaluation. Distillation is orthogonal to block length and could be applied to our model as well. (The Emilia model was trained after submission.)
- Time to first audio is at most 20 ms for every model here: ours delivers a 0.64 s block after 18 ms, the frame-level models an 80 ms frame after 8 to 13 ms.
DiTAR has no released code or weights, so it cannot be evaluated or timed under our protocol; we will add its reported numbers to Table 5, marked as reported. It reports 2.39 % WER and 0.67 SIM on the same test list (0.6B parameters, about 100k h, 10 steps per 100 ms patch) and a real-time factor of 0.66 at batch 1 on an A100, where ours is 0.006 on a B300 with text encoding and decoding included: about 100 times lower, more than the hardware difference accounts for. Our matched flow-matching baseline is a DiTAR-style model on our data and latents (a causal backbone with a local diffusion transformer run for many steps per patch), which lets us compare one step against many with everything else fixed.
Re: Listener ratings (Q2)
We ran a controlled listening study: 55 raters on Prolific (headphone screening and attention checks), 39 utterances, one per test speaker, each sample played after the speaker prompt and rated from 1 to 5 on naturalness, audio quality and voice similarity (165 ratings per system and scale), plus 330 paired comparisons of the one-step model with and without the critic (CMOS, −3 to +3). "No critic" is the matched control throughout: the same fine-tuning without the critic. Intervals are 95 % bootstrap intervals over raters.
| system | Naturalness | Audio quality | Voice similarity |
|---|---|---|---|
| Human speech | 4.31 [4.14, 4.47] | 4.41 [4.25, 4.56] | 4.33 [4.16, 4.49] |
| VoXtream2 | 4.12 [3.93, 4.31] | 4.55 [4.41, 4.67] | 4.65 [4.55, 4.76] |
| F5-TTS v1, 16 steps | 3.90 [3.64, 4.13] | 3.97 [3.76, 4.17] | 4.54 [4.40, 4.68] |
| Pocket TTS recipe retrained on LibriSpeech (24 layers) | 3.59 [3.34, 3.84] | 3.34 [3.13, 3.56] | 4.09 [3.92, 4.25] |
| Ours trained on Emilia, critic, 1 step | 3.52 [3.32, 3.73] | 3.32 [3.10, 3.55] | 3.90 [3.68, 4.12] |
| Ours, critic, 1 step | 3.33 [3.08, 3.59] | 3.19 [2.99, 3.40] | 3.55 [3.35, 3.75] |
| Matched flow matching, 16 steps (48 evaluations per block) | 3.30 [3.09, 3.52] | 3.28 [3.08, 3.50] | 3.46 [3.24, 3.68] |
| Ours, no critic, 1 step | 3.16 [2.92, 3.41] | 2.83 [2.68, 2.99] | 3.50 [3.26, 3.73] |
Critic against no critic, paired preference: +0.48 [+0.33, +0.64], p < 0.001.
We observe the following.
- Listeners clearly prefer the model with the critic and rate its audio quality higher (+0.36 [+0.18, +0.55]), with voice similarity unchanged. This is the critic's role in our method: in the revised recipe speaker guidance supplies the similarity, and the critic restores the quality that guidance costs at one step.
- One step matches a 16-step flow-matching model trained with the same architecture, data and guidance construction, which uses 48 evaluations per block (differences of +0.03, −0.09 and +0.09; none significant). The speed gain therefore preserves naturalness and voice identity relative to many-step sampling.
- F5-TTS and VoXtream2 are rated higher. F5-TTS is not a streaming model: its first audio arrives after 135 ms, and after 1.3 s at batch 16, against 18 ms and 37 ms for ours. VoXtream2 is 14 times slower at batch 16. Training our recipe on Emilia narrows the gap (+0.35 on voice similarity). The retrained Pocket TTS recipe is rated higher on voice similarity and naturalness than our block-8, one-step model, the setting the study rated; the automatic metric shows the same ordering there (0.439 against 0.396). The block-4, two-step setting matches Pocket TTS on that metric (0.443; above).
In the revision, these human ratings will be the primary evidence for quality, in particular across systems built on different codecs and vocoders. The automatic metrics agree with listeners within our model family (the critic raises PQ by 0.5 and rated audio quality by 0.36) and will play a supporting role in the controlled ablations.
Re: An independent speaker-similarity evaluation (Q2)
We re-scored the submission's models with an ECAPA-TDNN speaker encoder trained on VoxCeleb (SpeechBrain), which shares nothing with WavLM.
| submission's model, one step | SIM (WavLM-based) | SIM (ECAPA) |
|---|---|---|
| no critic | 0.327 | 0.327 |
| critic | 0.366 | 0.372 |
We observe that the critic's similarity gain holds under the independent encoder (+0.045 [+0.031, +0.058], paired, speaker-cluster bootstrap), so it is not produced by the shared WavLM features. We will add an ECAPA column to Table 5.
We thank the reviewer for their review. We have run the corresponding experiments: Pocket TTS under our quality and timing protocol, including a version retrained on our data; a controlled listening study; a speaker-similarity evaluation that does not use WavLM; and a second test set.
Since submission we have also retrained our model with guidance on the speaker prompt, built into the one-step target in the same way as text guidance (the "revised recipe"). On the same 960 h, speaker similarity rises from 0.366 in the submission to 0.396 at one step, 0.420 with a second step, and 0.443 with block 4 and two steps, which matches Pocket TTS retrained on the same data (0.439) at 1.7 times its throughput. As with strong text guidance, the added guidance costs one-step quality on its own, and the critic stage corrects this (PQ 6.08 to 6.58, WER 3.27 % to 2.91 %). Unless noted, "ours" in the results below is this revised recipe with the additional speaker prompt guidance.
Re: The contribution
Existing one-step streaming speech generators produce at most about 170 ms of audio per network evaluation: one 80 ms frame in CALM and Pocket TTS, and a patch of up to about 170 ms in ARDiT, which is obtained by distilling a multi-step teacher. Our paper shows how to generate 640 ms per evaluation, trained from scratch without a teacher; on the same hardware this gives 3 to 4.4 times the throughput of a frame-level model of the same size (below). It also identifies what makes this hard and what resolves it. Its contributions are methodological. (1) A guided MeanFlow target that makes classifier-free guidance usable in one evaluation without a teacher; two-pass guidance fails at one step, and under a shared guidance sweep our target is better than the iMF recursive target at one step at every guidance strength. (2) An account of what makes few-step generation hard: flow curvature rises with block size, and one-step quality falls with it. (3) An analysis of the artifacts that strong guidance introduces in speech, and a short critic stage that corrects them at almost no cost (a small fraction of training, no inference cost, no teacher). Together these are what make large blocks usable, and listeners rate the resulting one-step model on par with a matched flow-matching model that uses 48 evaluations per block.
The study is deliberately controlled: every model is trained from scratch on LibriSpeech 960 h with the same latents and architecture, which is what allows the paper's matched ablations and analysis within our compute budget. The paper does not aim at state-of-the-art quality, which at present depends largely on data and compute, and it states openly that our models trail systems trained on far more data (Table 5). As validation that the findings transfer, we trained the same recipe on Emilia: intelligibility and speaker similarity improve, and performance has not saturated with our compute budget on this larger dataset (below).