acceptodds
Under review as a conference paper at ICLR 2027

CaDiS: Causal Diffusion Transformer for Streaming Speech Generation

Abstract

Diffusion and flow-matching models have advanced high-fidelity speech generation. However, their bidirectional attention requires generating a full utterance before playback, and iterative sampling adds further delay. Most streaming systems therefore rely on autoregressive generators and use diffusion only for acoustic decoding. In this paper, we ask instead whether a pretrained offline diffusion model can generate speech chunk by chunk in real time. We present **CaDiS**, an offline-to-streaming framework built on a causal diffusion transformer that decouples the architectural and acceleration challenges into a three-stage recipe: autoregressive diffusion adaptation finetunes the bidirectional model with block-causal attention to generate each chunk from past speech alone; causal consistency distillation reduces sampling to four steps; and distribution matching refinement restores teacher-level quality. We adapt CaDiS to two streaming tasks, text-to-speech (TTS) and voice conversion (VC), with dedicated task-specific efforts: a normalized positional encoding for asynchronously aligned text in TTS, and content-vector distillation plus reference-anchored positional re-encoding to address the voice consistency and artifact challenges in VC. We evaluate CaDiS on objective and subjective metrics across natural human and expressive anime voices. CaDiS preserves offline-teacher quality while streaming: its TTS model attains a ms first-packet latency and real-time factor on an A100 GPU, while its VC model achieves the best intelligibility and perceptual quality among streaming baselines with no look-ahead. Moreover, CaDiS attains among the best overall streaming efficiency profiles, demonstrating the potential of diffusion speech generators in streaming scenarios.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
diffusion models, streaming speech generation, diffusion distillation, text-to-speech, voice conversion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_s5WVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes CaDiS, a framework that adapts pretrained offline diffusion speech models for streaming TTS and VC. The method combines block-causal adaptation, consistency distillation to reduce generation to four denoising steps, and distribution matching to recover quality. It also incorporates task-specific designs to support text–speech alignment and speaker consistency during streaming generation.

Critical strengths
  1. The proposed method is tested on two different tasks, TTS and VC. The ability to handle both tasks shows the practical value of adapting an offline diffusion model for streaming
  2. Human evaluation shows strong English voice conversion quality. Further assessment covers multiple languages, expressive anime voice synthesis, long-form audio generation, and formal listening studies.
Critical weaknesses
  1. The proposed method is a combination of a three-stage recipe from Casual Forcing, using RAPR from earlier work and keeping the offline teacher's normalized positional encoding. It is difficult to understand convolution beyond these previous methods.
  2. Given the architecture the main question is whether the speech-specific adaptatons add value.
  3. The RAPR ablation does not isolate the contribution of positional re-encoding.
  4. CaDiS shows competitive performance on different tasks; however, I feel the comparisons under matched operating conditions are missing. Although the authors discuss the trade-offs between quality and latency, CaDiS’s VC first-packet latency is 575 ms, while X-VC’s is 285 ms (Table 4). Further, the English WER improves from 3.41% to 3.09%, but speaker similarity drops from 0.748 to 0.670, while perceptual quality improves significantly (Table 2). These results show a useful trade-off. Still, comparing the systems at the same latency would better highlight the practical advantage.
Questions for authors
  1. Can you provide CI for the main listening study? It would also help distinguish clear improvements from small differences in mean scores.
5 comments
AuthorsOpenReview ↗

Response to Reviewer s5WV

We thank the reviewer for the thoughtful and constructive review, and for recognizing the practical value of studying both TTS and VC and the quality of our English VC results. We agree that the distinction from prior work, the evidence for the speech-specific designs, and the quality–latency comparison should be made clearer. We address these points below.

CW1: Thank you for pointing this out. We agree that the three-stage recipe and RAPR build on prior work, and that the TTS positional encoding is retained from the offline teacher rather than introduced as a new encoding.

We want to emphasize that our contribution is adapting the general distillation framework Causal Forcing++ with speech-specific design and its supporting experimental results, rather than a new general-purpose distillation algorithm. We study this conversion for two different conditioning structures: asynchronous text–speech alignment in TTS and frame-synchronous content conditioning in VC.

For long-form VC, the RAPR configuration provides a concrete example of this adaptation. The original StreamAvatar paper uses a small reference sink with a relatively long history window in order to achieve better long-form generation capabilities, we use latent frames from a 10-second speaker reference together with a short rolling history to enable consistent performance in streaming generation. This configuration retains a target-speaker anchor while limiting the amount of generated acoustic history carried forward during streaming.

We have revised these sections in the paper accordingly. We appreciate the reviewer's suggestion to separate these inherited components from the speech-specific adaptation and its supporting evidence more explicitly.

CW2: We agree that the speech-specific design choices should be supported by targeted experiments, rather than only by the performance of the complete system. The stage-wise results in Table 5 characterize the overall conversion pipeline, but do not by themselves establish the contribution of every task-specific choice.

For TTS, we compared retaining the offline teacher’s normalized positional encoding with an alternative design using RoPE and explicit sentence-level length conditioning. We trained an additional streaming TTS model using the same three-stage pipeline, replacing normalized positional encoding with RoPE and providing the total number of acoustic latent frames (num_frames) through adaptive layer normalization. During training, this number is derived from the training audio; during inference, it is provided by the duration estimator. The results on SeedTTS-eval English are shown below.

MethodWER (%) ↓WeSpeaker SIM ↑UTMOS ↑FPL (ms) ↓RTF ↓
Normalized positional encoding (ours, paper)1.690.7854.192860.31
RoPE + num_frames2.340.7834.273060.34

CaDiS achieves lower WER, slightly higher speaker similarity, and slightly lower FPL and RTF, whereas the RoPE variant achieves higher UTMOS. These results show a trade-off rather than uniform superiority of either design. They suggest that, for full-text-ready streaming TTS, explicitly conditioning the model on the total sequence length is a viable alternative to incorporating this information through normalized positional encoding. We have therefore clarified that normalized positional encoding is retained from the offline teacher, rather than introduced as a new encoding or claimed to be necessary or uniformly superior to RoPE.

AuthorsOpenReview ↗

Response to Reviewer s5WV (2)

CW3: We are grateful to the reviewer for pointing this out. We agree that changing the history-window length does not isolate the effect of positional re-encoding: it also changes the acoustic context available to the model.

Our comparison keeps the speaker reference, retained-history length, chunk size, and sampling budget fixed, and changes only whether the retained history and current chunk are positionally re-anchored. Both settings use a 1 s history window, 400 ms generator/ContentVec chunks, and four sampling steps. The monotonic control also undergoes Stage-3 distribution matching refinement.

Short-form English VC.

SystemWER (%) ↓WeSpeaker SIM ↑UTMOS ↑
Monotonic, 1 s history2.900.6743.94
RAPR, 1 s history2.830.6753.93

The first table shows the short-form evaluation on SeedTTS-VC. The two settings have close performance on all metrics. The paired WER/SIM confidence intervals cross zero, so these results do not establish a short-form advantage or equivalence.

Long-form English VC: 20 paired sources, mean duration 124.2 s, without sentence resets. Details in Section 4.1 and Appendix.

SystemFull-clip WER (%) ↓Early target SIM ↑Late target SIM ↑Late − early SIM
Monotonic, 1 s history15.390.6190.125−0.494
RAPR, 1 s history9.180.6990.696−0.003

However, the long-form evaluation in the second table, evaluated on 20 VC sources, shows that without re-anchoring, target-speaker similarity drops much more quickly than with RAPR, even after Stage-3 refinement training using the monotonic-position scheme. These experiments support the use of RAPR's bounded positional re-anchoring for long-form VC. Early/late SIM is the mean WeSpeaker similarity over non-overlapping 10-s windows whose centers fall within the first/last 30 s of each clip.

CW4: We agree that the submitted results show a useful trade-off rather than uniform superiority. Thank you for highlighting this. Although our model is trained on fixed 400 ms chunk size, it is capable of streaming in variable chunk sizes. We have used our streaming VC checkpoint with smaller generator and ContentVec chunks to approach the latency of X-VC and other baseline systems.

We evaluate 400 ms, 200 ms, and 100 ms chunks using the same checkpoint, four steps, and 1 s RAPR history. The results are shown below.

Generator chunk sizeWER (%) ↓WeSpeaker SIM ↑UTMOS ↑FPL (ms) ↓RTF ↓
100 ms3.420.6713.722721.10
200 ms3.270.6753.853730.59
400 ms2.850.6753.935760.33

For 100 ms chunks, the FPL approaches X-VC's 285 ms result, but the RTF increases above 1, so this setting does not establish sustained real-time conversion at matched latency. Different chunk sizes show a quality–latency trade-off: smaller chunks reduce FPL, while WER and UTMOS worsen and speaker similarity changes relatively little. For practical use, the 200 ms setting offers lower startup latency with RTF < 1, at some quality cost. We thank the reviewer for suggesting this more practically meaningful comparison.

AuthorsOpenReview ↗

Response to Reviewer s5WV (3)

Q1: We agree that confidence intervals, particularly for between-system differences, are needed to distinguish clear improvements from small differences in mean scores. We now provide 95% CIs for the main English listening-study means and paired differences, using 20,000 assignment-cluster bootstrap resamples.

For VC versus X-VC, the paired QMOS difference is +0.8869 [0.7143, 1.0625], while the SMOS difference is −0.0603 [−0.2270, 0.1034], from our listening study data. These results support a meaningful quality–similarity trade-off: the QMOS interval is entirely above zero, while the paired SMOS difference is not resolved at the reported pointwise 95% level.

The tables below provide the main English listening-study means and 95% confidence intervals, keeping the reported means unchanged.

Main Table 1: TTS means and 95% CIs

SystemNMOS [95% CI]QMOS [95% CI]SMOS [95% CI]
F5-TTS3.78 [3.67, 3.90]4.01 [3.89, 4.12]3.79 [3.67, 3.92]
IndexTTS-2.53.62 [3.50, 3.73]3.80 [3.68, 3.92]3.68 [3.54, 3.82]
S1-mini3.65 [3.51, 3.77]4.15 [4.04, 4.26]3.34 [3.19, 3.49]
Qwen3-TTS3.91 [3.80, 4.01]4.09 [3.98, 4.19]3.69 [3.57, 3.81]
VoXtream23.62 [3.50, 3.73]3.92 [3.81, 4.03]3.78 [3.65, 3.91]
CosyVoice33.51 [3.38, 3.64]4.00 [3.89, 4.12]3.68 [3.54, 3.82]
Faster-IndexTTS-23.75 [3.64, 3.86]3.82 [3.70, 3.95]3.80 [3.67, 3.93]
Ours offline3.90 [3.80, 4.01]4.17 [4.06, 4.27]3.85 [3.74, 3.97]
CaDiS TTS (1000-step)3.99 [3.86, 4.12]4.17 [4.04, 4.29]3.79 [3.65, 3.94]

Main Table 2: VC means and 95% CIs

SystemNMOS [95% CI]QMOS [95% CI]SMOS [95% CI]Rhy [95% CI]Exp [95% CI]
Vevo3.96 [3.82, 4.10]3.95 [3.82, 4.08]3.67 [3.52, 3.82]3.95 [3.78, 4.11]3.71 [3.56, 3.86]
Seed-VC3.32 [3.16, 3.47]2.99 [2.80, 3.17]2.90 [2.74, 3.06]3.65 [3.48, 3.83]3.43 [3.25, 3.61]
X-VC3.40 [3.23, 3.57]2.84 [2.66, 3.02]3.16 [3.01, 3.32]3.85 [3.67, 4.02]3.64 [3.47, 3.80]
MeanVC-23.58 [3.44, 3.72]3.26 [3.08, 3.43]2.37 [2.22, 2.52]3.73 [3.55, 3.90]3.52 [3.36, 3.69]
Ours offline3.92 [3.78, 4.06]3.87 [3.73, 4.01]3.01 [2.85, 3.17]3.95 [3.78, 4.12]3.81 [3.66, 3.95]
CaDiS VC3.79 [3.65, 3.93]3.73 [3.58, 3.87]3.10 [2.95, 3.26]4.00 [3.84, 4.15]3.82 [3.67, 3.96]

We sincerely thank the reviewer for providing a comprehensive review of our paper. We have added all the experiments discussed in this rebuttal to the Appendix of the paper, with blue text color showing the revision from submission version. We will consider moving some of the experiments to the main content upon acceptance.

Reviewer_oXzHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CaDiS turns a pretrained 0.8B offline DiT speech generator (flow matching on 40 Hz DAC-VAE latents, 48 kHz) into a block-causal, 4-step streaming model. The recipe follows Causal Forcing++ from video: block-causal fine-tuning with teacher forcing, causal consistency distillation, then DMD on self-rollouts. TTS keeps the teacher's normalized positions via a predicted utterance length; VC uses a causal 40 Hz ContentVec plus RAPR (fixed speaker reference, short rolling history).

Critical strengths
  1. It answers its question convincingly for English. The stage-wise ablation (Table 5) shows a small cost from causalization, a clear drop at 4 steps, and recovery after only 1k DMD steps, with the same pattern for TTS and VC. English VC gets the best WER, UTMOS and QMOS among streaming systems.
  2. The evaluation is careful and transparent: all baselines rerun with shared scoring, three similarity models, Mandarin, anime and long-form sets, and a well-designed listening study. The RAPR attention analysis (Table 18, Fig. 2) explains speaker drift instead of jut measuring it.
Critical weaknesses

Points 1 and 2 drive my score.

  1. The method is mostly a transfer, and "among the first streamable DiT speech generators on continuous latents" is overstated. The three stages come from Causal Forcing++ and RAPR from StreamAvatar. ARDiT and DiTAR already generate continuous speech latents autoregressively, and other continuous autoregressive generators such as VibeVoice (ICLR 2026), FELLE and AudioCALM(NeurIPS 2026) aren't cited or discussed. The paper's own premise, that converting an offline model is better than training a causal one, is never tested.
  2. The latency comparison isn't apples to apples. CosyVoice3 at 1553 ms suggests it was run without its streaming setup, and the inference engine and settings for each baseline aren't given. TTS needs the whole sentence and a predicted length before it starts, unlike full-stream systems such as VoXtream2. For VC, "zero look-ahead" comes with a fixed 400 ms chunk and the highest FPL (575 ms), there's no chunk-size ablation, and it isn't stated whether DAC-VAE decoding is causal per chunk. So "among the best overall streaming efficiency" is too strong.
  3. The TTS-specific design is barely examined. Normalized positions aren't compared with resetting or relative positions, the duration predictor isn't described at all, and long-form TTS stability mostly comes from resetting at every sentence rather than from the model.
  4. Speaker similarity is the consistent weak spot, and the headline metric hides it. With Seed-TTS-eval's standard WavLM-SV, CaDiS has the lowest TTS similarity among streaming systems (0.647, Table 6). In VC its SMOS is below X-VC on every benchmark, and the offline VC teacher is already far below Vevo (0.653 vs 0.756). Outside English the picture is mixed too: Anim400K TTS naturalness and quality MOS are below Qwen3-TTS and CosyVoice3, and Mandarin TTS CER is the highest among streaming systems, which the appendix calls "mid-pack".
  5. Smaller things. The RAPR window and step count are picked on the test set (Tables 17, 19), the main tables give no CIs for between-system MOS differences, and with 311k hours of internal data and no stated release, reproduction is hard.
Questions for authors
  1. For each baseline, which inference engine, acceleration and chunk settings did you use? What does each reported FPL include (text front end, duration prediction, first chunk, VAE decode)?
  2. Is DAC-VAE decoding causal per chunk, or does it use overlap or future latents?
  3. How accurate is the duration predictor, and do normalized positions actually beat a reset or relative scheme?
  4. Do you have any evidence that converting an offline model beats training a block-causal model directly at similar compute?
5 comments
AuthorsOpenReview ↗

Response to Reviewer oXzH

We thank the reviewer for the thoughtful review and the positive assessment of the English results and evaluation design. We agree that novelty, operating conditions, and the limits of the results should be stated more precisely.

CW1 / Q4: We agree that the original phrase was too broad. Our distinction is that CaDiS converts pretrained bidirectional speech DiTs into few-step streaming generators over continuous audio latents, with the DiT itself serving as the acoustic generative backbone, without a separate autoregressive speech LM.

DiTAR combines a separate causal LM with a local diffusion transformer, whereas CaDiS converts the pretrained speech DiT itself into the streaming acoustic backbone. ARDiT is a diffusion-backbone model operating on mel-autoencoder latents; our study addresses conversion of available bidirectional speech DiTs over audio latents for separate TTS and VC models. We expand the discussion of FELLE, VibeVoice, and AudioCALM.

We agree that the three-stage recipe and RAPR are built on prior work, and that the TTS positional encoding is retained from the offline teacher rather than introduced as a new encoding. Our contribution is the speech-specific adaptation and its supporting evidence, rather than a new general-purpose distillation algorithm. We study this conversion for two different conditioning structures: asynchronous text–speech alignment in TTS and frame-synchronous content conditioning in VC. We have revised these sections in the paper accordingly.

We do not currently have experimental evidence that offline-to-streaming conversion outperforms direct block-causal training at matched compute. Table 5 evaluates the stages within our conversion pipeline, rather than comparing these two training directions.

AuthorsOpenReview ↗

Response to Reviewer oXzH (2)

CW2 / Q1–Q2: We thank the reviewer for these concerns. Complete-sentence TTS and incremental-input TTS impose different waiting requirements.

All systems were evaluated with the complete target text available before timing started, including incremental-capable baselines such as VoXtream2. Thus, the comparison uses a consistent sentence-available input condition, rather than mixing full-sentence and incremental-input measurements. CaDiS's duration estimate is already included in the first-chunk timing path, and text-arrival waiting is zero under this setup. Our comparison therefore concerns full-text-ready synthesis, not incremental-text capabilities.

The original CosyVoice3 efficiency measurement already used stream=True and its official vLLM backend. However, after warm-up, its output hop remained at 100, so acoustic decoding waited for the speech-token prefix required by this larger hop. We remeasured independent requests with an initial hop of 25, preserving the native 25→50→100 within-request growth, 25-token acoustic chunk, and 3-token flow look-ahead, following the official implementation. The updated measurement gives 923 ms FPL and 0.34 RTF. But this doesn't affect our conclusions drawn in Section 4.4 on efficiency comparison.

SystemEngine / accelerationChunk / context (ms of audio)
Qwen3-TTSfaster-qwen3-tts; CUDA graphs640
VoXtream2Native CUDA graphs; Mimi80
CosyVoice3 — follow-upOfficial vLLM; FP32 flow/vocoder1000 chunk; hop 1000→2000→4000; 120 look-ahead
Faster-IndexTTS-2TensorRT / TensorRT-LLM≈2000 window; ≈93 overlap
Seed-VCPyTorch250 block + 50 crossfade + 20 right context
X-VCPyTorch120 current + 20 smoothing + 100 future
MeanVC-2PyTorch120 current + 40 future
CaDiS–TTSPyTorch400
CaDiS–VCPyTorch400; no future look-ahead

The table gives the engine and chunk settings for each baseline. For TTS, timing includes the text front end, first-chunk generation, and waveform decoding; CaDiS also includes duration estimation. Model loading and warm-up are excluded. The faster-qwen3-tts for Qwen3-TTS engine is from https://github.com/andimarafioti/faster-qwen3-tts.

Zero additional look-ahead in VC does not eliminate the 400 ms block buffer when computing FPL because the computation of FPL in VC includes waiting for the first chunk to arrive, the model processing time and the decoding time.

DAC-VAE decoding is not sample-causal; we use the same offline DAC-VAE model as in the offline setting. Streaming decoding is achieved by crossfading between decoded latent chunks.

Although our model is trained on fixed 400 ms chunk size, it is capable of streaming in variable chunk sizes. We have used our streaming VC checkpoint with smaller generator chunk sizes to approach the latency of X-VC and other baseline systems. We evaluate 400 ms, 200 ms, and 100 ms chunks using the same checkpoint, four steps, and 1 s RAPR history. The results are shown below.

Generator chunk sizeWER (%) ↓WeSpeaker SIM ↑UTMOS ↑FPL (ms) ↓RTF ↓
100 ms3.420.6713.722721.10
200 ms3.270.6753.853730.59
400 ms2.850.6753.935760.33

For 100 ms chunks, the FPL approaches X-VC's 285 ms result, but the RTF increases above 1, so this setting does not establish sustained real-time conversion at matched latency. Different chunk sizes show a quality–latency trade-off: smaller chunks reduce FPL, while WER and UTMOS worsen and speaker similarity changes relatively little. For practical use, the 200 ms setting offers lower startup latency with RTF < 1, at some quality cost. We thank the reviewer for suggesting this more practically meaningful comparison.

AuthorsOpenReview ↗

Response to Reviewer oXzH (3)

CW3 / Q3: We agree that the speech-specific design choices should be supported by targeted experiments, rather than only by the performance of the complete system. The stage-wise results in Table 5 characterize the overall conversion pipeline, but do not by themselves establish the contribution of every task-specific choice.

For TTS, we compared retaining the offline teacher’s normalized positional encoding with an alternative design using RoPE and explicit sentence-level length conditioning. We trained an additional streaming TTS model using the same three-stage pipeline, replacing normalized positional encoding with RoPE and providing the total number of acoustic latent frames (num_frames) through adaptive layer normalization. During training, this number is derived from the training audio; during inference, it is provided by the duration estimator. The results on SeedTTS-eval English are shown below.

MethodWER (%) ↓WeSpeaker SIM ↑UTMOS ↑FPL (ms) ↓RTF ↓
Normalized positional encoding (ours, paper)1.690.7854.192860.31
RoPE + num_frames2.340.7834.273060.34

CaDiS achieves lower WER, slightly higher speaker similarity, and slightly lower FPL and RTF, whereas the RoPE variant achieves higher UTMOS. These results show a trade-off rather than uniform superiority of either design. They suggest that, for full-text-ready streaming TTS, explicitly conditioning the model on the total sequence length is a viable alternative to incorporating this information through normalized positional encoding. We have therefore clarified that normalized positional encoding is retained from the offline teacher, rather than introduced as a new encoding or claimed to be necessary or uniformly superior to RoPE.

The duration estimator is a reference-rate heuristic, not a separately trained neural network. We derive the average number of acoustic latent frames per IPA token from the reference audio and its transcript, then use this rate to estimate the target sentence duration, with adjustments for punctuation and non-speech events. The reported long-form TTS system also resets at sentence boundaries.

CW4: We agree that these weaknesses deserve explicit discussion. The subjective results on SeedTTS-eval English shows that in this experiment, the wespeaker similarity aligns better with human judgement, and since we have subjective studies conducted, we can focus more on the subjective results on speaker similarity (i.e. SMOS). The offline VC teacher is already far below Vevo on SeedTTS-eval VC task doesn't affect our claim that our adaptation framework can preserve the teacher's quality in this case. Regarding the superior speaker similarity score of CaDiS VS X-VC, we argue that we achieve a significantly better trade off as we have a better audio quality, as X-VC tends to trade audio quality for better speaker similarity, with the audio quality not very usable in real life. We agree that the word mid-pack is misused in Appendix. We have changed it in the paper. Our model training data only consists of a very small amount of Chinese data, so we didn't get very good results in Chinese, but the results can still prove that this adaptation framework can preserve the offline teacher's quality on the metrics.

We would like to emphasize that speaker similarity should be considered together with intelligibility and perceptual quality. On SeedTTS-eval English VC, CaDiS achieves a QMOS of 3.73 versus 2.84 for X-VC. The paired QMOS difference is +0.8869 [0.7143, 1.0625], whereas the SMOS difference is −0.0603 [−0.2270, 0.1034] from our listening study data. These results support a meaningful quality–similarity trade-off: the QMOS interval is entirely above zero, while the paired SMOS difference is not resolved at the reported pointwise 95% level.

The offline teacher's gap to Vevo should also be distinguished from the effect of streaming adaptation. The former characterizes the starting generator; the latter requires a teacher–student comparison. On English TTS, the QMOS point estimate remains 4.17 for both models. On Anim400K TTS, WER changes from 1.14% to 1.17%, although QMOS and SMOS decrease. We therefore describe preservation in a metric- and benchmark-dependent sense, while acknowledging the remaining perceptual-quality gaps in English VC and Anim400K TTS.

Reviewer_meYGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CaDiS, a framework for adapting a pre-trained DiT to streaming speech generation. It employs a 3-stage training process consisting of (1) Autoregressive Diffusion Adaptation, (2) Causal Consistency Distillation, and (3) Distribution Matching Refinement. The same DiT backbone is used for both TTS and VC, with separate task-specific models designed to maintain text-speech alignment and speaker consistency in streaming settings. Experimental results show that CaDiS achieves generation quality close to its offline teacher with only 4 sampling steps.

Critical strengths

S1. This paper explores a practical way to adapt an offline diffusion model for streaming speech generation. Enabling block-causal generation without training a bi-directional DiT from scratch is a reasonable and efficient approach.

S2. The experiments are well structured. In particular, Table 5 clearly shows the effect of each stage in the 3-stage training pipeline, including the quality degradation from fewer sampling steps and the improvement after DMD.

Critical weaknesses

W1. The task-specific design choices for TTS and VC lack sufficient supporting ablations. In Section 3.4 and introduction, the authors explain that normalized positional encoding is important for maintaining text-speech alignment during chunk-wise generation. However, there is no ablation study that removes this encoding or compares it with alternative methods. The paper also mentions using an external duration predictor to estimate the total utterance duration, but provides no detail about how this component is implemented.

W2. The streaming efficiency comparison is not entirely fair, as the systems operate under different input conditions. CaDiS-TTS requires the full-sentence text and an estimated utterance duration before synthesis, whereas systems such as VoXtream2 support incremental text input. This makes the reported FPL numbers difficult to compare directly, especially in interactive streaming scenarios. In addition, although the CaDiS-VC achieves zero look-ahead, its FPL is 575 ms, which is the highest among the compared systems.

W3. Some of the claims seem stronger than what the experimental results support. On Anim400K TTS datset, both QMOS and SMOS decrease compared with the offline teacher. For VC, the speaker similarity are also consistently lower than those of X-VC. Given these results, the claims about preserving offline teacher quality and achieving strong overall VC performance should be toned down.

W4. The paper provides limited details about the training data and experimental setup. The authors report using up to 311K and 179K hours of training data for TTS and VC, respectively. However, the dataset composition, original audio sampling rates (sr), and preprocessing procedures are not specified. The audio samples on the CaDiS demo page appear to be provided at 48 kHz. Since the compared systems have different output sr, it is unclear whether the authors applied re-sampling or other normalization to ensure a consistent evaluation setup. Also, while the offline DiT teacher is reported to have 0.8B parameters, the total parameter count of the complete system is not provided.

Questions for authors

Q1. Are all training datasets sampled at 48 kHz? Was 48 kHz also used when measuring RTF?

Q2. Could the authors compare RAPR with standard sliding-window attention under the same context length?

Q3. How can the FPL of full-sentence TTS and incremental-input TTS be compared under a consistent evaluation setup? How would the E2E latency change if duration prediction and the time spent waiting for text input were also taken into account?

5 comments
AuthorsOpenReview ↗

Response to Reviewer meYG

We thank the reviewer for the careful assessment and for recognizing the practical value of offline-to-streaming adaptation and the stage-wise experiments. We agree that the task-specific evidence, latency comparison, and implementation details need clearer explanation.

CW1: We agree that the complete-system results do not establish the necessity of normalized positional encoding. The stage-wise results in Table 5 characterize the overall conversion pipeline, but do not by themselves establish the contribution of every task-specific design choice.

To investigate this question, we trained an additional streaming TTS model using the same three-stage pipeline, replacing normalized positional encoding with RoPE. To retain sentence-level length conditioning, we also provided the total number of acoustic latent frames (num_frames) through adaptive layer normalization. During training, this number is derived from the training audio; during inference, it is provided by the duration estimator. The results on SeedTTS-eval English are shown below.

MethodWER (%) ↓WeSpeaker SIM ↑UTMOS ↑FPL (ms) ↓RTF ↓
Normalized positional encoding (ours, paper)1.690.7854.192860.31
RoPE + num_frames2.340.7834.273060.34

CaDiS achieves lower WER, slightly higher speaker similarity, and slightly lower FPL and RTF, whereas the RoPE variant achieves higher UTMOS. These results show a trade-off rather than uniform superiority of either design. They suggest that, for full-text-ready streaming TTS, explicitly conditioning the model on the total sequence length is a viable alternative to incorporating this information through normalized positional encoding. We have therefore revised the paper to clarify that we retain normalized positional encoding from the offline teacher, rather than claim that it is necessary or uniformly superior to RoPE.

The duration estimator is a reference-rate heuristic, not a separately trained neural network. We derive the average number of acoustic latent frames per IPA token from the reference audio and its transcript, and use this rate to estimate the target sentence duration, with adjustments for punctuation and non-speech events. The reported long-form TTS system also resets at sentence boundaries.

AuthorsOpenReview ↗

Response to Reviewer meYG (2)

CW2 / Q3: We thank the reviewer for highlighting this distinction. CaDiS-TTS streams audio from complete-sentence text; it is not incremental-input TTS.

All systems were evaluated with the complete target text available before timing started, including incremental-capable baselines such as VoXtream2. Thus, the comparison uses a consistent sentence-available input condition, rather than mixing full-sentence and incremental-input measurements. CaDiS's duration estimate is already included in the first-chunk timing path, and text-arrival waiting is zero under this setup. Our comparison therefore concerns full-text-ready synthesis, not incremental-text capabilities.

The original CosyVoice3 efficiency measurement already used stream=True and its official vLLM backend. However, after warm-up, its output hop remained at 100, so acoustic decoding waited for the speech-token prefix required by this larger hop. We remeasured independent requests with an initial hop of 25, preserving the native 25→50→100 within-request growth, 25-token acoustic chunk, and 3-token flow look-ahead, following the official implementation. The updated measurement gives 923 ms FPL and 0.34 RTF. But this doesn't affect our conclusions drawn in Section 4.4 on efficiency comparison.

SystemEngine / accelerationChunk / context (ms of audio)
Qwen3-TTSfaster-qwen3-tts; CUDA graphs640
VoXtream2Native CUDA graphs; Mimi80
CosyVoice3 — follow-upOfficial vLLM; FP32 flow/vocoder1000 chunk; hop 1000→2000→4000; 120 look-ahead
Faster-IndexTTS-2TensorRT / TensorRT-LLM≈2000 window; ≈93 overlap
Seed-VCPyTorch250 block + 50 crossfade + 20 right context
X-VCPyTorch120 current + 20 smoothing + 100 future
MeanVC-2PyTorch120 current + 40 future
CaDiS–TTSPyTorch400
CaDiS–VCPyTorch400; no future look-ahead

The table gives the engine and chunk settings for each baseline. For TTS, timing includes the text front end, first-chunk generation, and waveform decoding; CaDiS also includes duration estimation. Model loading and warm-up are excluded. The faster-qwen3-tts for Qwen3-TTS engine are from https://github.com/andimarafioti/faster-qwen3-tts.

For VC, zero additional look-ahead does not remove chunk buffering, and the reported 575 ms FPL is higher than those of the compared baselines.

We have used our streaming VC checkpoint with smaller generator chunk sizes to approach the latency of X-VC and other baseline systems. We evaluate 400 ms, 200 ms, and 100 ms chunks using the same checkpoint, four steps, and 1 s RAPR history. The results are shown below.

Generator chunk sizeWER (%) ↓WeSpeaker SIM ↑UTMOS ↑FPL (ms) ↓RTF ↓
100 ms3.420.6713.722721.10
200 ms3.270.6753.853730.59
400 ms2.850.6753.935760.33

For 100 ms chunks, the FPL approaches X-VC's 285 ms result, but the RTF increases above 1, so this setting does not establish sustained real-time conversion at matched latency. Different chunk sizes show a quality–latency trade-off: smaller chunks reduce FPL, while WER and UTMOS worsen and speaker similarity changes relatively little. For practical use, the 200 ms setting offers lower startup latency with RTF < 1, at some quality cost. We thank the reviewer for suggesting this more practically meaningful comparison.

AuthorsOpenReview ↗

Response to Reviewer meYG (3)

CW3: Thank you; we agree that the claims need qualification.

We would like to emphasize that speaker similarity should be considered together with intelligibility and perceptual quality. On SeedTTS-eval English VC, CaDiS achieves a QMOS of 3.73 versus 2.84 for X-VC. The paired QMOS difference is +0.8869 [0.7143, 1.0625], whereas the SMOS difference is −0.0603 [−0.2270, 0.1034] from our listening study data. These results support a meaningful quality–similarity trade-off: the QMOS interval is entirely above zero, while the paired SMOS difference is not resolved at the reported pointwise 95% level.

The offline teacher's gap to Vevo should also be distinguished from the effect of streaming adaptation. The former characterizes the starting generator; the latter requires a teacher–student comparison. On English TTS, the QMOS point estimate remains 4.17 for both models. On Anim400K TTS, WER changes from 1.14% to 1.17%, although QMOS and SMOS decrease. We therefore describe preservation in a metric- and benchmark-dependent sense, while acknowledging the remaining perceptual-quality gaps in English VC and Anim400K TTS.

We also provide 95% confidence intervals for the main English listening-study means and paired system differences, keeping the reported means unchanged.

Main Table 1: TTS means and 95% CIs

SystemNMOS [95% CI]QMOS [95% CI]SMOS [95% CI]
F5-TTS3.78 [3.67, 3.90]4.01 [3.89, 4.12]3.79 [3.67, 3.92]
IndexTTS-2.53.62 [3.50, 3.73]3.80 [3.68, 3.92]3.68 [3.54, 3.82]
S1-mini3.65 [3.51, 3.77]4.15 [4.04, 4.26]3.34 [3.19, 3.49]
Qwen3-TTS3.91 [3.80, 4.01]4.09 [3.98, 4.19]3.69 [3.57, 3.81]
VoXtream23.62 [3.50, 3.73]3.92 [3.81, 4.03]3.78 [3.65, 3.91]
CosyVoice33.51 [3.38, 3.64]4.00 [3.89, 4.12]3.68 [3.54, 3.82]
Faster-IndexTTS-23.75 [3.64, 3.86]3.82 [3.70, 3.95]3.80 [3.67, 3.93]
Ours offline3.90 [3.80, 4.01]4.17 [4.06, 4.27]3.85 [3.74, 3.97]
CaDiS TTS3.99 [3.86, 4.12]4.17 [4.04, 4.29]3.79 [3.65, 3.94]

Main Table 2: VC means and 95% CIs

SystemNMOS [95% CI]QMOS [95% CI]SMOS [95% CI]Rhy [95% CI]Exp [95% CI]
Vevo3.96 [3.82, 4.10]3.95 [3.82, 4.08]3.67 [3.52, 3.82]3.95 [3.78, 4.11]3.71 [3.56, 3.86]
Seed-VC3.32 [3.16, 3.47]2.99 [2.80, 3.17]2.90 [2.74, 3.06]3.65 [3.48, 3.83]3.43 [3.25, 3.61]
X-VC3.40 [3.23, 3.57]2.84 [2.66, 3.02]3.16 [3.01, 3.32]3.85 [3.67, 4.02]3.64 [3.47, 3.80]
MeanVC-23.58 [3.44, 3.72]3.26 [3.08, 3.43]2.37 [2.22, 2.52]3.73 [3.55, 3.90]3.52 [3.36, 3.69]
Ours offline3.92 [3.78, 4.06]3.87 [3.73, 4.01]3.01 [2.85, 3.17]3.95 [3.78, 4.12]3.81 [3.66, 3.95]
CaDiS VC3.79 [3.65, 3.93]3.73 [3.58, 3.87]3.10 [2.95, 3.26]4.00 [3.84, 4.15]3.82 [3.67, 3.96]

CW4 / Q1: We agree that these details are important for interpreting and reproducing the results. Most data from our datasets are 48kHz audio, and for data with a sample rate lower than 48kHz, we upsample them to 48kHz before feeding in to the model, and our generator produces 48 kHz audio, which is also used when measuring its RTF. Baseline timing retains each system's native output rate. For the supplementary evaluations, we normalize the scoring audio to −23 LUFS with a 0.95 peak ceiling and use the fixed metric-specific resampling pipeline. We used our internal data and due to our data policy, we cannot reveal more details about the dataset, but we can enclose more details if possible within the timeline. The complete TTS system has about 1.5B parameters, combining the ~0.8B DiT, 564M mT5 text encoder, and 128M DAC-VAE. The VC system has about 0.65B parameters, combining the 421M DiT, 94M ContentVec encoder, and the same DAC-VAE. These totals include frozen components.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.