acceptodds
Under review as a conference paper at ICLR 2027

MeanFlow for One-Step Streaming Text-to-Speech

Abstract

A text-to-speech service must start speaking soon after each request arrives while batching many requests on each GPU. Non-autoregressive flow-matching models denoise the whole utterance at every step, so they stream nothing until they finish and their latency grows with the batch; autoregressive models stream, but generating token by token takes many sequential steps or cascades of models. We study a simpler design: a single causal model that generates speech as blocks of continuous codec latents, each in one to three network evaluations, trained from scratch with a MeanFlow objective. We find that the block size trades generation difficulty for speed, that classifier-free guidance must be built into the training target and pushes the output away from real speech, and that a short fine-tuning stage with a critic in a pretrained speech representation improves one-step speaker similarity and quality at every block size at no inference cost. Trained on LibriSpeech, the fine-tuned model achieves near-human intelligibility in one step, produces its first audio in under 20 ms, and delivers more than four times the peak throughput of recent flow-matching models at their fastest intelligible settings.

Primary area:
generative models
Keywords:
TTS, speech, meanflow, flow matching, diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_haycOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper builds a streaming TTS model that generates blocks of continuous speech latents with one to three network evaluations per block. It adapts MeanFlow training to include text guidance and adds a short critic fine-tuning stage to improve one-step output. On LibriSpeech-PC, the model shows low word error rates, an 18 ms time to first audio, and higher throughput than the tested systems. Speaker similarity remains well below the stronger baselines. (Sections 3–8; Table 5.)

Critical strengths
  1. The blockwise design gives a clear reduction in startup delay and improves throughput as the batch grows.
  2. The guidance and critic ablations are useful, including a control that gets the same extra training without the critic.
Critical weaknesses
  1. MeanFlow and patchwise continuous speech generation already exist. The guided target is useful, but Table 5 omits DiTAR and CALM/Pocket TTS, which Section 2 identifies as close designs. The matched flow-matching control establishes the benefit of fewer steps, but does not establish an advantage over comparable few-step streaming systems. This leaves the size of the contribution uncertain.

  2. Evaluation is limited to LibriSpeech-PC test-clean, and no aggregate listener ratings are reported. Table 5 gives speaker similarity of 0.366, versus 0.554 for F5-TTS after the same codec round trip. Both the critic and the similarity metric also rely on WavLM. The automatic quality scores therefore leave open whether the speed gain preserves satisfactory voice identity and naturalness.

Questions for authors

Were DiTAR or CALM/Pocket TTS evaluated under the same timing and quality protocol? If results are available, where does this model fall on their speed-quality trade-off?

Does the listening bundle have collected listener ratings, or is there an independent speaker-similarity evaluation that supports the critic’s gains beyond the WavLM-based metric?

5 comments

We thank the reviewer for their review. We have run the corresponding experiments: Pocket TTS under our quality and timing protocol, including a version retrained on our data; a controlled listening study; a speaker-similarity evaluation that does not use WavLM; and a second test set.

Since submission we have also retrained our model with guidance on the speaker prompt, built into the one-step target in the same way as text guidance (the "revised recipe"). On the same 960 h, speaker similarity rises from 0.366 in the submission to 0.396 at one step, 0.420 with a second step, and 0.443 with block 4 and two steps, which matches Pocket TTS retrained on the same data (0.439) at 1.7 times its throughput. As with strong text guidance, the added guidance costs one-step quality on its own, and the critic stage corrects this (PQ 6.08 to 6.58, WER 3.27 % to 2.91 %). Unless noted, "ours" in the results below is this revised recipe with the additional speaker prompt guidance.

Re: The contribution

Existing one-step streaming speech generators produce at most about 170 ms of audio per network evaluation: one 80 ms frame in CALM and Pocket TTS, and a patch of up to about 170 ms in ARDiT, which is obtained by distilling a multi-step teacher. Our paper shows how to generate 640 ms per evaluation, trained from scratch without a teacher; on the same hardware this gives 3 to 4.4 times the throughput of a frame-level model of the same size (below). It also identifies what makes this hard and what resolves it. Its contributions are methodological. (1) A guided MeanFlow target that makes classifier-free guidance usable in one evaluation without a teacher; two-pass guidance fails at one step, and under a shared guidance sweep our target is better than the iMF recursive target at one step at every guidance strength. (2) An account of what makes few-step generation hard: flow curvature rises with block size, and one-step quality falls with it. (3) An analysis of the artifacts that strong guidance introduces in speech, and a short critic stage that corrects them at almost no cost (a small fraction of training, no inference cost, no teacher). Together these are what make large blocks usable, and listeners rate the resulting one-step model on par with a matched flow-matching model that uses 48 evaluations per block.

The study is deliberately controlled: every model is trained from scratch on LibriSpeech 960 h with the same latents and architecture, which is what allows the paper's matched ablations and analysis within our compute budget. The paper does not aim at state-of-the-art quality, which at present depends largely on data and compute, and it states openly that our models trail systems trained on far more data (Table 5). As validation that the findings transfer, we trained the same recipe on Emilia: intelligibility and speaker similarity improve, and performance has not saturated with our compute budget on this larger dataset (below).

Re: Comparison with CALM / Pocket TTS and DiTAR (Q1)

We thank the reviewer for this question. In short: on matched data our model matches the Pocket TTS recipe on the speaker-similarity metric at 1.7 times its throughput (block 4, two steps), or runs 3 to 4.4 times faster at somewhat lower similarity (block 8, one step).

How the two designs relate. Pocket TTS works in the same latent space as our model (Mimi) but generates one 80 ms frame per step, where ours generates a block of eight frames (640 ms). In the terms of our paper it has a block size of 1. Our block-size analysis (Fig. 5, block sizes 4 to 32) shows that shorter blocks are more accurate in one step and slower, because each second of audio takes more network evaluations; Pocket TTS sits past the short end of that range. Our paper addresses the opposite end: how to generate a long block in a single evaluation, which is where the throughput comes from and what the guided target and the critic are for.

What we compared. The released Pocket TTS checkpoint is a 6-layer student distilled from a 24-layer model trained on 88k h, and the 24-layer model is not released. To compare on matched data, we used Kyutai's training code to retrain their recipe (the 24-layer model, their sampler defaults) on LibriSpeech 960 h. Speed is measured on one B300 under the paper's timing protocol (Appendix D).

matched data: LibriSpeech 960 haudio per stepWER %SIMPQaudio-s / s, batch 1batch 16
Pocket TTS recipe (24 layers), retrained80 ms2.150.4396.6339352
Ours, block 4, 2 steps320 ms2.760.4436.4267601
Ours, block 8, 1 step640 ms2.910.3966.581721058
Ours, block 8, 2 steps640 ms2.880.4206.42125888
Ours, block 16, 1 step1.28 s4.230.3576.493041416
larger dataaudio per stepWER %SIMPQaudio-s / s, batch 1batch 16
Pocket TTS, released (distilled 6-layer student; 88k h)80 ms1.550.4936.751331036
Ours trained on Emilia (137k h, only ~2 epochs), block 8, 1 step640 ms2.190.4556.401721058
Ours trained on Emilia (137k h, only ~2 epochs), block 8, 2 steps640 ms2.000.4876.21125888

We observe the following.

  • Matched data. Block 4 with two steps reaches a similarity of 0.443, against 0.439 for the Pocket TTS recipe, while generating four times more audio per step and running 1.7 times faster. The Pocket TTS recipe keeps a lower WER (2.15 % against 2.76 %).
  • Block size is a dial. Longer blocks trade similarity for speed. At one step, similarity is 0.420, 0.396 and 0.357 for blocks 4, 8 and 16, and block 8 is 3 to 4.4 times faster than the Pocket TTS recipe. Pocket TTS, at one frame per step, is consistent with this trend.
  • Larger data. Trained on Emilia, our two-step model reaches 0.487, against 0.493 for the released Pocket TTS. That model gets its speed from distillation into a network a quarter of the size (90M parameters; ours has about 330M, the size of their 24-layer model). Ours reaches comparable speed without distillation, because it generates 0.64 s per evaluation. Distillation is orthogonal to block length and could be applied to our model as well. (The Emilia model was trained after submission.)
  • Time to first audio is at most 20 ms for every model here: ours delivers a 0.64 s block after 18 ms, the frame-level models an 80 ms frame after 8 to 13 ms.

DiTAR has no released code or weights, so it cannot be evaluated or timed under our protocol; we will add its reported numbers to Table 5, marked as reported. It reports 2.39 % WER and 0.67 SIM on the same test list (0.6B parameters, about 100k h, 10 steps per 100 ms patch) and a real-time factor of 0.66 at batch 1 on an A100, where ours is 0.006 on a B300 with text encoding and decoding included: about 100 times lower, more than the hardware difference accounts for. Our matched flow-matching baseline is a DiTAR-style model on our data and latents (a causal backbone with a local diffusion transformer run for many steps per patch), which lets us compare one step against many with everything else fixed.

Re: Listener ratings (Q2)

We ran a controlled listening study: 55 raters on Prolific (headphone screening and attention checks), 39 utterances, one per test speaker, each sample played after the speaker prompt and rated from 1 to 5 on naturalness, audio quality and voice similarity (165 ratings per system and scale), plus 330 paired comparisons of the one-step model with and without the critic (CMOS, −3 to +3). "No critic" is the matched control throughout: the same fine-tuning without the critic. Intervals are 95 % bootstrap intervals over raters.

systemNaturalnessAudio qualityVoice similarity
Human speech4.31 [4.14, 4.47]4.41 [4.25, 4.56]4.33 [4.16, 4.49]
VoXtream24.12 [3.93, 4.31]4.55 [4.41, 4.67]4.65 [4.55, 4.76]
F5-TTS v1, 16 steps3.90 [3.64, 4.13]3.97 [3.76, 4.17]4.54 [4.40, 4.68]
Pocket TTS recipe retrained on LibriSpeech (24 layers)3.59 [3.34, 3.84]3.34 [3.13, 3.56]4.09 [3.92, 4.25]
Ours trained on Emilia, critic, 1 step3.52 [3.32, 3.73]3.32 [3.10, 3.55]3.90 [3.68, 4.12]
Ours, critic, 1 step3.33 [3.08, 3.59]3.19 [2.99, 3.40]3.55 [3.35, 3.75]
Matched flow matching, 16 steps (48 evaluations per block)3.30 [3.09, 3.52]3.28 [3.08, 3.50]3.46 [3.24, 3.68]
Ours, no critic, 1 step3.16 [2.92, 3.41]2.83 [2.68, 2.99]3.50 [3.26, 3.73]

Critic against no critic, paired preference: +0.48 [+0.33, +0.64], p < 0.001.

We observe the following.

  • Listeners clearly prefer the model with the critic and rate its audio quality higher (+0.36 [+0.18, +0.55]), with voice similarity unchanged. This is the critic's role in our method: in the revised recipe speaker guidance supplies the similarity, and the critic restores the quality that guidance costs at one step.
  • One step matches a 16-step flow-matching model trained with the same architecture, data and guidance construction, which uses 48 evaluations per block (differences of +0.03, −0.09 and +0.09; none significant). The speed gain therefore preserves naturalness and voice identity relative to many-step sampling.
  • F5-TTS and VoXtream2 are rated higher. F5-TTS is not a streaming model: its first audio arrives after 135 ms, and after 1.3 s at batch 16, against 18 ms and 37 ms for ours. VoXtream2 is 14 times slower at batch 16. Training our recipe on Emilia narrows the gap (+0.35 on voice similarity). The retrained Pocket TTS recipe is rated higher on voice similarity and naturalness than our block-8, one-step model, the setting the study rated; the automatic metric shows the same ordering there (0.439 against 0.396). The block-4, two-step setting matches Pocket TTS on that metric (0.443; above).

In the revision, these human ratings will be the primary evidence for quality, in particular across systems built on different codecs and vocoders. The automatic metrics agree with listeners within our model family (the critic raises PQ by 0.5 and rated audio quality by 0.36) and will play a supporting role in the controlled ablations.

Re: An independent speaker-similarity evaluation (Q2)

We re-scored the submission's models with an ECAPA-TDNN speaker encoder trained on VoxCeleb (SpeechBrain), which shares nothing with WavLM.

submission's model, one stepSIM (WavLM-based)SIM (ECAPA)
no critic0.3270.327
critic0.3660.372

We observe that the critic's similarity gain holds under the independent encoder (+0.045 [+0.031, +0.058], paired, speaker-cluster bootstrap), so it is not produced by the shared WavLM features. We will add an ECAPA column to Table 5.

Reviewer_uRDwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies streaming text-to-speech generation using autoregressive blocks of continuous audio latents. A causal encoder provides text and speech-history conditioning, while a MeanFlow decoder generates each block in a small number of network evaluations. The method modifies the guidance target used in iMF by constructing it from zero-guidance predictions, and subsequently applies adversarial fine-tuning with a critic operating on pretrained speech representations. The experiments examine generation quality, latency, throughput, and the effects of block size and sampling steps.

Critical strengths
  1. The paper addresses a practical problem: combining low first-audio latency with efficient generation under batching. The evaluation across block sizes and sampling steps helps characterize the resulting quality–efficiency trade-off.
  2. The critic fine-tuning experiments include a control with the same additional training budget. This is useful for distinguishing the contribution of adversarial supervision from the benefit of simply training longer.
Critical weaknesses
  1. Unclear iMF baseline setup. The paper does not clarify whether the iMF baseline directly supervises zero-span predictions at nonzero guidance, which its recursive target queries. Simply inheriting the proposed method’s zero-guidance FM partition could disadvantage this baseline.

  2. Guidance strengths are not controlled. Table 3 uses standard CFG scales of 3.5 for the proposed method and 2 for iMF, without matched-strength comparisons or a clear common tuning protocol. The reported improvement therefore cannot be attributed solely to the target modification.

Questions for authors
  1. For the iMF baseline in Table 3, what are the exact zero-span/nonzero-span sampling proportions and guidance distributions? In particular, are zero-span predictions at nonzero guidance directly supervised?
  2. How were the guidance strengths in Table 3 selected? Can the authors provide a shared guidance sweep for both methods under comparable training and tuning budgets?
  3. Does the proposed target retain its advantage when the zero-span supervision and other training settings are controlled?
3 comments

We thank the reviewer for the careful questions about the guidance comparison. We are happy to clarify the baseline and to provide the requested sweep, which confirms the one-step result of Table 3 at every guidance strength.

Re: How the iMF baseline is trained

Yes: in the baseline, zero-span predictions at nonzero guidance are directly supervised. It does not inherit our zero-guidance flow-matching partition.

We implemented the baseline from the iMF paper and its official code. There, the network takes the guidance scale as an input, and every training example, zero-span or not, is trained toward the guided velocity at its sampled scale. That target combines the network's own prediction with the text at that scale and its unguided prediction without the text. Our baseline does the same, with the guidance scale drawn from iMF's power-law distribution.

Our method differs in exactly this respect. Zero-span examples are trained as plain flow matching with the guidance input set to zero, and guidance enters only through the nonzero-span examples (CFG scale drawn uniformly from 1 to 6), whose target is built from unguided predictions.

Everything else is held at our setting, so that the two models differ only in the guidance target and the sampling of its scale: 87.5 % zero-span and 12.5 % nonzero-span rows per batch, the instantaneous velocity read from the model's own boundary prediction (one of the two options in the iMF paper), and the same data, architecture, batch size, training steps and span sampler. We will add these details to Appendix B.

Re: A shared guidance sweep

In Table 3 each model was evaluated at the middle of the guidance distribution it was trained on: CFG scale 3.5 for ours and 2 for iMF. We thank the reviewer for suggesting a shared sweep, which makes the comparison much clearer. Both models of Table 3 are evaluated on one grid, CFG scales 1 to 5, at one step. Entries are WER % / SIM / PQ on LibriSpeech-PC.

CFG scaleoursiMF target
19.5 / 0.282 / 6.0623.0 / 0.216 / 5.71
1.56.4 / 0.298 / 6.2214.3 / 0.238 / 5.84
26.3 / 0.296 / 6.3211.4 / 0.242 / 5.89
2.55.7 / 0.292 / 6.3310.8 / 0.237 / 5.87
35.5 / 0.290 / 6.3010.5 / 0.235 / 5.81
3.55.8 / 0.284 / 6.2510.2 / 0.229 / 5.74
45.3 / 0.282 / 6.1510.5 / 0.228 / 5.66
55.5 / 0.275 / 6.039.9 / 0.216 / 5.50

We observe that at one step our target is better on all three metrics at every guidance strength. With each model at its own best strength, WER is 5.3 % against 9.9 %, with higher speaker similarity (0.30 against 0.24) and higher PQ. The result of Table 3 is therefore not a consequence of the strengths chosen: no guidance strength closes the gap. We will add the sweep to the paper.

Re: Does the advantage hold under matched settings?

Yes. In this comparison the baseline is trained with its own zero-span supervision (the guided target on every row), and all other settings are shared. At one step our target is ahead at every guidance strength. Our reading is that the two targets differ in what they are built from. In ours, the zero-span examples train the unguided velocity directly against data, and the guided target is a fixed combination of those well-supervised predictions. In the recursive target, the network's own guided prediction appears inside its regression target: the target moves as the network trains, errors in the guided prediction feed back into it, and the guided field is only reached as a fixed point. This matters most at one step, where there are no further steps to correct the prediction.

In light of the baseline details and the shared sweep, we would like to ask whether the reviewer would be willing to reconsider their score. We are happy to discuss any of these points further.

As the discussion period is ending soon, we wanted to check whether our response above has addressed your concerns. In it, we clarify the iMF baseline implementation, which we adapt from the official iMF code. We also add the shared guidance sweep you requested, in which our target is ahead on WER, SIM and PQ at every strength.

If any concerns remain, we would be glad to discuss them in the remaining time.

Thank you for clarifying how the iMF baseline was trained and for comparing the two methods at the same guidance strengths. These additional results give me more confidence in the reported one-step performance. I would raise my assessment of the paper from 2 to 2.5.

Reviewer_nDN5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a blockwise continuous autoregressive (AR) text-to-speech (TTS) model trained from scratch using an Improved MeanFlow (iMF) objective to enable low-latency, high-throughput speech synthesis. Instead of performing frame-by-frame token generation or full-utterance non-autoregressive flow matching, the proposed architecture generates continuous audio VAE latents in fixed-size blocks using one to three network function evaluations (NFE) per block. To address the quality degradation of single-step generation, the authors apply a short adversarial fine-tuning phase. The paper provides extensive ablation studies analyzing the impact of individual design choices, including block size, guidance scale and construction, the number of sampling steps, and critic fine-tuning. Empirically, the fine-tuned block-8 model matches the WER of their codec-resynthesized recordings at 1 NFE per block, while achieving a time to first audio (TTFA).

Critical strengths
  • Practical few-step streaming TTS with a favorable speed–quality trade-off. The paper applies MeanFlow blockwise in a single causal model. Compared with the same architecture trained with many-step flow matching, the one-step model fine-tuned with the critic is on par in intelligibility and quality and better in speaker similarity, at much lower latency and higher throughput.

  • Thorough ablations against matched baselines, with insightful analysis. The training objective, the guidance construction and the critic are each compared with a matched alternative. The analysis also explains the results, such as why few-step generation gets harder as blocks grow and how guidance distorts the output spectrum.

Critical weaknesses
  • Incremental novelty and unacknowledged baselines. The core components (iMF, blockwise or patch-wise autoregressive generation of continuous latents, and adversarial fine-tuning of few-step generators) are established techniques. ARDiT [1], for example, already generates each block of continuous latents in one NFE. SplitMeanFlow, IntMeanFlow and DiTAR are cited only briefly and without numerical comparison. Yet DiTAR reports results on the same LibriSpeech-PC test set, and Appendix A describes SplitMeanFlow arms whose results are not reported. ARDiT and CosyVoice 2 [2] are not cited, and Pocket TTS appears only in related work. Pocket TTS is the most directly comparable system, since it generates the same Mimi VAE latents frame by frame and has a public checkpoint. CosyVoice 2, an open-weight streaming system that pairs a language model with chunk-aware flow matching, would be a more representative streaming baseline than VoXtream2 alone.

  • Low performance. One-step WER (3.19%) remains above F5-TTS (1.91–2.50%). Matching codec resynthesis is a weak reference. Relying on codec resynthesis as a matching baseline is problematic: adopting a lower-capacity VAE (the compact Mimi VAE from Pocket-TTS) and attributing degraded reference scores to codec limitations cannot be fully justified. Speaker similarity (0.31–0.37) is also too low to blame on the codec. With the same VAE and protocol, CALM reports 0.52 against a resynthesis ceiling of 0.57, albeit with far more training data.

  • Limited data scale. All models are trained only on LibriSpeech (960 h), roughly 40–100 times less data than the open baselines. The paper attributes its speaker-similarity gap to data scale and to the missing speaker-prompt guidance but tests neither. It is therefore unclear whether blockwise one-step generation itself limits similarity. A joint text and speaker-prompt guidance ablation, or a single larger-data run, would separate these factors.

  • Heavy reliance on adversarial critic fine-tuning. Against the matched no-critic control, the critic lowers one-step WER from 4.06% to 3.19% and raises SIM from 0.327 to 0.366. SplitMeanFlow and IntMeanFlow avoid an adversarial stage by using a flow-matching teacher, and CALM, or Pocket TTS, trains its one-step head from scratch with neither. Since the critic stage is short, the authors should discuss when it is preferable to teacher-based distillation.

References:

[1] Z. Liu, S. Wang, S. Inoue, Q. Bai, and H. Li. Autoregressive Diffusion Transformer for Text-to-Speech Synthesis. arXiv:2406.05551, 2024.

[2] Z. Du et al. CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models. arXiv:2412.10117, 2024.

Questions for authors

Most of my questions are raised in the Critical Weaknesses above.

5 comments

We thank the reviewer for the detailed review and the pointers to prior work. We have run the experiments the reviewer proposed (guidance on the speaker prompt, a larger-data run) and added the baselines the reviewer names (Pocket TTS, CosyVoice 2, SplitMeanFlow), with quality and speed measured under one protocol.

Since submission we have retrained our model with guidance on the speaker prompt, built into the one-step target in the same way as text guidance (the "revised recipe"; details under "What limits speaker similarity"). Unless noted, "ours" in the results below is this revised recipe with the critic.

Re: Novelty and relation to prior work

Existing one-step streaming speech generators produce at most about 170 ms of audio per network evaluation: one 80 ms frame in CALM and Pocket TTS, and a patch of up to about 170 ms in ARDiT, which is obtained by distilling a multi-step teacher. Our paper shows how to generate 640 ms per evaluation, trained from scratch without a teacher; on the same hardware this gives 3 to 4.4 times the throughput of a frame-level model of the same size (below). It also identifies what makes this hard and what resolves it. Its contributions are methodological and conceptual, and they hold independently of data scale.

  1. A guided one-step target without a teacher. Two-pass guidance cannot be applied to a one-step jump (126.8 % WER in Table 3), so guidance has to be part of the training target. Our target is built from unguided branches. Under a shared guidance sweep it is better than the iMF recursive target at one step on every metric and at every guidance strength (5.3 % against 9.9 % WER and 0.30 against 0.24 SIM, each at its best strength). The same construction extends to a second guidance axis on the speaker prompt.
  2. Block size determines the difficulty of few-step generation. Flow curvature rises with block size, and one-step quality falls with it (Fig. 5). Shorter blocks are easier to generate in one step and cost throughput.
  3. An analysis of guidance artifacts in speech, and a critic stage that corrects them. Past a moderate strength, guidance leaves WER flat while similarity and quality fall, with a measurable spectral excess. A short critic stage corrects this at almost no cost (a small fraction of training, no inference cost, no teacher), and its feature space determines what it corrects (Table 4). Listeners confirm the effect (paired preference +0.48 [+0.33, +0.64] over the matched control, 330 pairs).
  4. A controlled comparison of one-step and many-step sampling. With architecture, data and guidance held fixed, one evaluation per block is rated by listeners on par with 16-step flow matching at 48 evaluations per block (within 0.1 on naturalness, audio quality and voice similarity, no difference significant; 55 raters).

The study is deliberately controlled: every model is trained from scratch on LibriSpeech 960 h with the same latents and architecture, which is what allows the matched ablations above within our compute budget. The paper does not aim at state-of-the-art quality, which at present depends largely on data and compute, and it states openly that our models trail systems trained on far more data (Table 5). The highest-scoring of those systems are also an order of magnitude slower at batch 16 (below). As validation that the findings transfer, we trained the same recipe on Emilia: intelligibility and speaker similarity improve, and performance has not saturated within our limited compute budget.

We will cite and discuss ARDiT. It generates blocks of continuous latents autoregressively with flow matching and then distils the 16-step model, with distribution matching distillation, so that a block of up to about 170 ms takes one evaluation.

Re: Baselines

Our protocol measures quality and speed on the same hardware, which requires released weights. We have added the two released systems the reviewer names. Speed is on one B300 under the paper's timing protocol (Appendix D: 5 s prompt, 15 s generated, the whole call inside the measurement).

LibriSpeech-PCtraining dataWER %SIMPQaudio-s / s, batch 1batch 16first audio
CosyVoice 2 (released, streaming)about 167k h2.010.6636.97857376 ms
F5-TTS v1 (Tables 5-6; non-streaming, 32 steps)95k h1.910.6756.395791263 ms
VoXtream2 (Tables 5-6; streaming)40k h3.320.6596.3057697 ms
Pocket TTS, released (distilled 6-layer student)88k h1.550.4936.7513310368 ms
Pocket TTS recipe (24 layers), retrained960 h2.150.4396.633935213 ms
Ours, block 4, 2 steps960 h2.760.4436.426760119 ms
Ours, block 8, 1 step960 h2.910.3966.58172105818 ms
Ours, block 8, 2 steps960 h2.880.4206.4212588820 ms
Ours, block 16, 1 step960 h4.230.3576.49304141617 ms
Ours trained on Emilia, block 8, 1 / 2 steps137k h, only ~2 epochs2.19 / 2.000.455 / 0.4876.40 / 6.21172 / 1251058 / 88818 / 20 ms

One pattern holds across the table. The three systems with the highest speaker similarity, F5-TTS, VoXtream2 and CosyVoice 2, are an order of magnitude slower at batch 16: they generate 91, 76 and 57 seconds of audio per second, against 1058 for our model, and their first audio arrives after 97 to 376 ms against 18 ms. Their higher quality comes with far more computation per second of audio. Our contribution is on the other side of that trade-off: the throughput and latency above come from generating a 0.64 s block in one step. Against the one system of comparable speed, the released Pocket TTS, our Emilia model is on par in speaker similarity (0.487 against 0.493) after only about two epochs of training.

CosyVoice 2 is timed in its fastest documented configuration (fp16, TensorRT, and vLLM for the language model, as in its own example script). It is the strongest system on quality, consistent with its data scale, and about 20 times slower than our model.

Pocket TTS / CALM works in the same latent space as our model (Mimi) but generates one 80 ms frame per step, where ours generates a block of eight frames (640 ms). In the terms of our paper it has a block size of 1. Our block-size analysis (Fig. 5, block sizes 4 to 32) shows that shorter blocks are more accurate in one step and slower, because each second of audio takes more network evaluations; Pocket TTS sits past the short end of that range. Our paper addresses the opposite end: how to generate a long block in a single evaluation, which is where the throughput comes from and what the guided target and the critic are for.

The CALM model the reviewer cites (313M parameters, SIM 0.52) is not released. The released Pocket TTS is a 6-layer student distilled from it, and scores 0.493 under our protocol. For a controlled comparison we retrained their recipe (the 24-layer model, their training code and sampler defaults) on LibriSpeech 960 h. We observe the following.

  • At matched data, our model matches the Pocket TTS recipe on the speaker-similarity metric at higher throughput. Block 4 with two steps reaches 0.443 against 0.439, generates 320 ms per step where the frame-level recipe generates 80 ms, and is 1.7 times faster at batch 1 and at batch 16. The frame-level recipe keeps a lower WER (2.15 % against 2.76 %).
  • Block size and step count are two dials on one system (Fig. 5): block 8 with one step is 3 to 4.4 times faster than the frame-level recipe at somewhat lower similarity (0.396), and block 16 is 1.3 to 1.8 times faster again at lower quality. At one step similarity is 0.357, 0.396 and 0.420 for blocks 16, 8 and 4, so the frame-level recipe's similarity is consistent with the trend in our own models.
  • With Emilia, a second step takes similarity from 0.455 to 0.487, against 0.493 for the released Pocket TTS. That model is a 90M-parameter student distilled from a 313M-parameter teacher. Ours has about 330M parameters, the size of their teacher, and runs at the speed of their student because it generates 0.64 s per evaluation; at equal size their recipe is 3 to 4.4 times slower. Distillation is orthogonal to block length and could be applied to our model as well.

SplitMeanFlow was proposed for a different setting. It's applied to the acoustic stage of a two-stage system (conditioned on speech tokens from a language model), and the few-step model is distilled from a pretrained flow-matching teacher. We reimplemented its interval-splitting objective in our setting, trained from scratch with the same architecture and data.

block 8, revised recipe, one step (WER % / SIM / PQ)ours (MeanFlow objective)interval-splitting objective
before the critic stage4.40 / 0.393 / 5.783.32 / 0.373 / 5.39

We observe that the interval-splitting objective gives good intelligibility, but with noticeable artifacts that show up as lower speaker similarity and markedly lower quality (PQ 5.39 against 5.78). With the critic stage, our objective's one-step WER is 2.91 %.

DiTAR has no released code or weights, so it cannot be evaluated or timed under our protocol; we will add its reported numbers to Table 5, marked as reported. It reports 2.39 % WER and 0.67 SIM on the same test list (0.6B parameters, about 100k h, 10 steps per 100 ms patch) and a real-time factor of 0.66 at batch 1 on an A100, where ours is 0.006 on a B300 with text encoding and decoding included: about 100 times lower, more than the hardware difference accounts for. Our matched flow-matching baseline is a DiTAR-style model on our data and latents (a causal backbone with a local diffusion transformer run for many steps per patch), which lets us compare one step against many with everything else fixed.

Re: What limits speaker similarity (data scale and speaker-prompt guidance)

The paper attributed the gap to data scale and to the absence of guidance on the speaker prompt. We thank the reviewer for proposing the two experiments that test this. We ran both, and both explanations hold.

Speaker-prompt guidance. We added a second guidance scale on the speech history, trained by dropping the history and built into the one-step target in the same way as text guidance, so sampling remains one network evaluation. The text and speaker scales are independent and can each be set at inference.

Data scale. We trained the same recipe on the English portion of Emilia (137k h; trained after submission). For compute reasons this model is under-trained relative to the open systems (about 2.4 epochs on two GPUs; F5-TTS, for example, reports 1.2M updates on eight GPUs, about 11 epochs), and its speaker similarity was still rising when training stopped.

LibriSpeech-PC (WER % / SIM / PQ)1 step2 steps
Text guidance only, no critic, 960 h3.12 / 0.358 / 6.532.79 / 0.367 / 6.70
Text + speaker-prompt guidance, critic (the revised recipe)2.91 / 0.396 / 6.582.88 / 0.420 / 6.42
Revised recipe, block 42.60 / 0.420 / 6.412.76 / 0.443 / 6.42
Revised recipe, Emilia 137k h (only ~2 epochs)2.19 / 0.455 / 6.402.00 / 0.487 / 6.21
Flow matching, 16 steps, same speaker guidance, 960 h5.06 / 0.399 / 6.31

We observe the following.

  • Both factors contribute. Speaker-prompt guidance adds about 0.04 to 0.05 and the larger corpus a further 0.06 to 0.07.
  • One-step sampling is not what limits similarity. A flow-matching model with the same architecture, data and guidance construction, sampled with 16 steps, reaches 0.399, against 0.396 for one step.
  • On matched data the model matches the closest system: block 4 with two steps reaches 0.443, against 0.439 for Pocket TTS retrained on the same 960 h. With both factors, the two-step model reaches 0.487, on par with the released Pocket TTS (0.493 under our protocol; the 0.52 the reviewer cites is for the larger CALM model, which is not released).
  • The recipe transfers and has not saturated: more data improves intelligibility and speaker similarity, and similarity was still rising when training stopped.

We will update the paper with these results.

Re: Performance and the codec reference

Codec resynthesis is reported as the reference for what this latent space preserves, alongside the recordings themselves: the recordings score 2.61 % WER and 0.695 SIM, and 3.12 % and 0.559 after the codec. The codec therefore accounts for part of the similarity gap but not all of it. F5-TTS shows the same drop through the same codec (0.675 to 0.554), and the paper attributes the remainder to the model, which the experiments above now confirm (speaker guidance and data scale). On intelligibility, the revised recipe’s one-step WER is 2.91 % on LibriSpeech and 2.19 % when trained on Emilia, against 2.61 % for the recordings and 1.91 % for F5-TTS, a non-streaming model that uses 32 sampling steps (64 network evaluations). Where the paper says “near-human intelligibility”, we will state these reference numbers.

Reviewer_MvQWOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a streaming TTS system that generates continuous Mimi codec latents block by block. A causal audio encoder represents the transcript and preceding speech, while a MeanFlow decoder generates the next block in one to three network evaluations. The paper further introduces a guidance-conditioned MeanFlow target built from zero-guidance conditional/unconditional branches and a short adversarial fine-tuning stage using a critic over frozen WavLM features. On LibriSpeech, the one-step block-8 model with critic fine-tuning reaches 3.19% WER, 18 ms batch-1 time to first audio, and substantially higher throughput than the evaluated flow-matching systems.

Critical strengths
  1. The paper gives a technically coherent route from MeanFlow to practical streaming TTS: blockwise continuous generation avoids token-level autoregression, the guided target makes one-step text guidance usable, and the encoder/decoder split confines the expensive JVP to the decoder. Training from scratch without a teacher makes the result more consequential than another distillation recipe.

  2. The empirical analysis is unusually careful. Block size, NFE, guidance construction/scale, and critic design are studied with matched controls; uncertainty is estimated by speaker-cluster bootstrap; and serving measurements are end-to-end on the same hardware. In particular, Table 5 supports the main efficiency result while transparently showing the remaining speaker-similarity gap.

Critical weaknesses
  1. The block-autoregressive system has an unaddressed train–inference mismatch. During training, every block is conditioned on clean history and blocks are denoised in parallel; the critic likewise places a generated block after two real blocks, and Appendix A explicitly disables history noising. At inference, however, each block conditions on previously generated speech. No generated-history training, long-form evaluation, boundary metric, or error-versus-block-position analysis is provided. Thus the evidence on relatively short LibriSpeech utterances and a fixed 15 s timing workload does not establish that quality, speaker identity, or latency remains stable over long streaming outputs. Full-history attention is also not accompanied by a caching or length-scaling analysis.

  2. The central perceptual claims rely entirely on automatic metrics. PQ is a learned non-reference predictor, and speaker similarity uses WavLM-based embeddings while the critic is itself optimized through frozen WavLM features; the latter creates evaluation-model coupling even if the exact heads differ. The provided listening bundle is not a controlled human study. Moreover, the system comparison omits the closest blockwise/few-step systems discussed in Related Work (e.g., DiTAR, SplitMeanFlow, StreamMel, or Pocket TTS). Consequently, the paper convincingly establishes computational efficiency, but not yet the claimed perceptual advantage or relative standing among the closest streaming approaches.

Questions for authors
  1. How do WER, speaker similarity, quality, and block-boundary artifacts evolve with block position on 30–60 s generations? Can the authors compare clean-history training with generated/noised-history training and let the critic observe generated context?
  2. Can the authors report a controlled human listening study for naturalness/quality and speaker similarity, including the one-step model before/after critic fine-tuning, the three-step model, and representative streaming and non-streaming baselines?
6 comments

We thank the reviewer for the careful reading and the concrete suggestions. We ran the requested experiments: a controlled listening study, an analysis by position and length, a noised-history ablation, a speaker-similarity evaluation that does not use WavLM, and comparisons with the closest released systems, including Pocket TTS retrained on our data.

Two points of context. First, the study is deliberately controlled: every model is trained from scratch on LibriSpeech 960 h with the same latents and architecture, which is what allows the paper's matched comparisons within our compute budget. The paper does not aim at state-of-the-art quality, which at present depends largely on data and compute, and it states openly that our models trail systems trained on far more data. Second, as validation that the findings transfer, we trained the same recipe on Emilia: intelligibility and speaker similarity improve (one-step WER 2.91 % to 2.19 %, SIM 0.396 to 0.455), and performance has not saturated within our compute budget.

In response to a reviewer, we have also retrained our model with guidance on the speaker prompt, built into the one-step target in the same way as text guidance (the "revised recipe"). On the same 960 h, speaker similarity rises from 0.366 in the submission to 0.396 at one step, 0.420 with a second step, and 0.443 with block 4 and two steps, which matches Pocket TTS retrained on the same data (0.439) at 1.7 times its throughput. As with strong text guidance, the added guidance costs one-step quality on its own, and the critic stage corrects this (PQ 6.08 to 6.58, WER 3.27 % to 2.91 %). Unless noted, "ours" in the results below is this revised recipe with the additional speaker prompt guidance.

Re: Human listening study

We thank the reviewer for this suggestion and ran the controlled study described. 55 raters on Prolific (first language English, no hearing difficulties; headphone screening and attention checks) rated 39 utterances, one per test speaker. Each sample is played after the speaker prompt and rated from 1 to 5 on naturalness, audio quality and voice similarity (165 ratings per system and scale). Each rater also compared six pairs of the one-step model with and without the critic (CMOS, −3 to +3; 330 pairs). "No critic" is the matched control throughout: the same fine-tuning without the critic. Intervals are 95 % bootstrap intervals over raters.

systemnetwork evaluationsNaturalnessAudio qualityVoice similarity
Human speech–4.31 [4.14, 4.47]4.41 [4.25, 4.56]4.33 [4.16, 4.49]
VoXtream2 (streaming, open)n/a (autoregressive codec LM)4.12 [3.93, 4.31]4.55 [4.41, 4.67]4.65 [4.55, 4.76]
F5-TTS v1, 16 steps (non-streaming, open)32 per utterance3.90 [3.64, 4.13]3.97 [3.76, 4.17]4.54 [4.40, 4.68]
Pocket TTS recipe retrained on LibriSpeech2 per 80 ms frame3.59 [3.34, 3.84]3.34 [3.13, 3.56]4.09 [3.92, 4.25]
Ours trained on Emilia, critic, 1 step1 per 640 ms block3.52 [3.32, 3.73]3.32 [3.10, 3.55]3.90 [3.68, 4.12]
Ours, critic, 1 step1 per block3.33 [3.08, 3.59]3.19 [2.99, 3.40]3.55 [3.35, 3.75]
Matched flow matching, 16 steps48 per block3.30 [3.09, 3.52]3.28 [3.08, 3.50]3.46 [3.24, 3.68]
Ours, no critic, 1 step1 per block3.16 [2.92, 3.41]2.83 [2.68, 2.99]3.50 [3.26, 3.73]

Critic against no critic, paired preference: +0.48 [+0.33, +0.64], p < 0.001.

We observe the following.

  • Listeners clearly prefer the model with the critic. On the scales it raises audio quality by +0.36 [+0.18, +0.55], with voice similarity unchanged within noise (+0.05 [−0.22, +0.34]). This is the critic's role in our method: in the revised recipe speaker guidance supplies the similarity, and the critic restores the quality that guidance costs at one step.
  • One step closes the gap to many-step sampling. The matched flow-matching model is trained separately with the same architecture, data and guidance construction and uses 48 network evaluations per block; the one-step model is within 0.1 of it on all three scales (+0.03, −0.09, +0.09; none significant). The study rated the one-step models, the paper's main setting, alongside five reference systems; the three-step model's automatic scores are in the table under "Train–inference mismatch".
  • F5-TTS and VoXtream2 are rated higher. F5-TTS is not a streaming model: its first audio arrives after 1.3 s at batch 16, against 37 ms for ours. VoXtream2 is 14 times slower at batch 16. Training our recipe on Emilia narrows the gap (+0.35 on voice similarity). The retrained Pocket TTS recipe is rated higher on voice similarity and naturalness than our block-8, one-step model, the setting the study rated; the automatic metric shows the same ordering there (0.439 against 0.396). The block-4, two-step setting, which matches Pocket TTS on that metric, is discussed under "the closest systems".

On the automatic quality metric: within our model family, PQ and listeners agree (the critic raises PQ by 0.5 and rated audio quality by 0.36, and is preferred in the paired test). Across systems built on different codecs and vocoders they do not rank systems in the same way. In the revision, human ratings will therefore be the primary evidence for quality, in particular across systems, with the automatic metrics in a supporting role for the controlled ablations. We will add the study to the paper.

Re: Evaluation-model coupling (WavLM in the critic and in SIM)

We re-scored the submission's models with an ECAPA-TDNN speaker encoder trained on VoxCeleb (SpeechBrain), which shares nothing with WavLM.

submission's model, one stepSIM (WavLM-based)SIM (ECAPA)
no critic0.3270.327
critic0.3660.372

We observe that the two encoders agree: the critic's similarity gain is +0.045 [+0.031, +0.058] under ECAPA, against +0.039 under the WavLM-based scorer, so it is not produced by the shared feature extractor. We will add an ECAPA column to Table 5.

Re: Train–inference mismatch

We thank the reviewer for raising this. We tested the standard remedy for the mismatch, noising the history during training (noise scale up to 0.5), on the submission's recipe and on the revised one.

block 8 (WER % / SIM / PQ)1 step2 steps3 steps
Submission recipe, clean history4.95 / 0.330 / 6.392.78 / 0.343 / 6.622.80 / 0.344 / 6.66
Submission recipe, noised history3.31 / 0.300 / 6.432.62 / 0.316 / 6.672.59 / 0.312 / 6.72
Revised recipe, clean history4.40 / 0.393 / 5.782.99 / 0.416 / 6.322.73 / 0.411 / 6.25
Revised recipe, noised history2.78 / 0.350 / 5.902.70 / 0.372 / 6.412.70 / 0.371 / 6.37
Revised recipe, clean history + critic2.91 / 0.396 / 6.582.88 / 0.420 / 6.422.71 / 0.413 / 6.30
Revised recipe, noised history + critic2.79 / 0.361 / 6.572.74 / 0.377 / 6.50

We observe that noising the history lowers the one-step WER before the critic stage (4.40 % to 2.78 %) and costs 0.03 to 0.04 of speaker similarity at every step count, because the model reads a degraded prompt. After the critic stage the WER difference is gone (2.91 % against 2.79 %) while the similarity cost remains. The fine-tuning stage therefore already removes the intelligibility errors that the mismatch causes, and clean-history training is the better recipe. We will add this ablation.

A critic that observes generated context is a natural extension, and we will discuss it as future work. Training, including the critic stage, processes all blocks of an utterance in parallel from the recorded history. A critic over generated context requires rolling the model out block by block inside each training step, with the critic's gradient carried back through the rollout, so the cost grows with the rollout length and the parallel training that keeps the method cheap is given up. The analysis below finds no accumulation of errors with position, so we expect the gain to be modest relative to that cost.

Re: Quality against position and length

Our models are trained on utterances of up to 20 s, which limits a single pass to roughly that length. Longer passages are generated sentence by sentence from the same prompt, the standard approach for models trained on utterance-length data (F5-TTS and Pocket TTS do the same). We report quality by position within a pass and across a 60 s passage.

Within a pass. Long test utterances (9 to 18 s targets), scored in 5 s windows (SIM of the window against the prompt / PQ / WER % of the reference words in that span):

positionrecordingsours, critic, 1 stepours, critic, 2 stepsours, no critic, 1 step
0–5 s0.670 / 6.54 / –0.389 / 6.39 / 3.80.420 / 6.27 / 3.00.398 / 5.91 / 3.2
5–10 s0.668 / 6.52 / –0.365 / 6.67 / 3.00.384 / 6.31 / 2.20.371 / 5.94 / 3.0

Beyond 10 s too few utterances remain to score similarity and PQ by window; the WER of the words in that span is 2.4 %, 2.8 % and 3.2 % for the three models.

Across a 60 s passage, chunked (consecutive test-clean sentences, each generated from the same prompt; ours with the critic, one step; two 5 s windows per cell):

position0–10 s10–20 s20–30 s30–40 s40–50 s50–55 s
SIM0.391, 0.3840.367, 0.3820.365, 0.3850.367, 0.3750.390, 0.3770.382
PQ6.46, 6.636.60, 6.506.74, 6.696.54, 6.626.55, 6.636.74
WER %4.0, 2.63.0, 3.33.3, 2.93.9, 3.03.6, 2.44.0

We observe that WER and PQ do not degrade with position within a pass, which is where errors could accumulate. In long-form generation each sentence starts again from the prompt, so errors cannot carry over from one sentence to the next, and quality is stable across the 60 s passage. Speaker similarity of a 5 s window falls by about 0.03 from the first window to the second. On the standard test set the same fall appears for the non-autoregressive F5-TTS (0.660 to 0.626) and for the released Pocket TTS (0.489 to 0.452), so it is not specific to our model or to conditioning on generated history. On block boundaries: we do not hear artifacts at the joins between blocks, and any such artifact would be reflected in the naturalness and audio-quality ratings of the listening study. We will add this analysis as an appendix.

Re: Latency against length (full-history attention)

The sequences involved are short: at 12.5 latent frames per second, 60 s of speech is 750 positions, small next to the contexts language models attend over. The cost of a single pass of each length, measured on one B200 with a 5 s prompt (block 8, one step):

generated lengthper-block latency, batch 1time to first audio, batch 1audio-s / s, batch 1audio-s / s, batch 16
5 s2.9 ms14 ms1571361
10 s2.8 ms14 ms1881447
20 s2.8 ms14 ms2011367
30 s3.1 ms17 ms1841196
60 s3.2 ms15 ms182871

We observe that the per-block latency is flat from 5 s to 60 s at batch 1 (2.8 to 3.2 ms per 0.64 s block). At batch 16 the model still generates 871 seconds of audio per second at 60 s; the decrease comes from re-attending to the history without a key-value cache. We will add this table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.