acceptodds
Under review as a conference paper at ICLR 2027

Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning

Abstract

Efficient zero-shot voice cloning remains challenging: autoregressive systems are slow, while many non-autoregressive systems still require reference transcripts during inference. We present Tacit-TTS, an efficient transcript-free zero-shot voice cloning system distilled from IndexTTS2. Tacit-TTS replaces autoregressive text-to-semantic decoding with masked non-autoregressive generation, introduces training-free acoustic length estimation, and accelerates the flow-matching renderer through ReFlow distillation. Across two English and two Mandarin datasets, Tacit-TTS achieves competitive zero-shot quality while generating speech over 10 times faster than IndexTTS2 for utterances longer than 5 seconds. Its transcript-free conditioning also supports cross-lingual and non-lexical references. We validate these settings using references from eight other languages, infant babble, and synthetic gibberish, where transcript-dependent systems often degrade or fail because of unreliable ASR transcripts, making reliable voice cloning difficult.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Zero-Shot Voice Cloning, Non-Autoregressive Text-to-Speech, Knowledge Distillation, Flow Matching, Masked Generative Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_JLKCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Tacit-TTS, an efficient transcript-free zero-shot voice cloning system distilled from IndexTTS2. The method replaces autoregressive text-to-semantic decoding with masked non-autoregressive generation, estimates output length from acoustic speaking-rate cues and target-text syllable counts, and accelerates the semantic-to-Mel renderer through ReFlow distillation. Experiments on English and Mandarin benchmarks demonstrate a favorable trade-off between generation speed and synthesis quality. Additional evaluations examine cross-lingual and non-lexical reference audio.

Critical strengths
  1. Clear practical motivation: Improving inference efficiency while retaining transcript-free reference conditioning addresses a useful deployment setting, particularly when reliable reference transcripts are unavailable.

  2. Promising empirical trade-off: The system substantially reduces generation latency while maintaining competitive intelligibility and speaker similarity. English speaker similarity is particularly strong, although content accuracy remains below the teacher.

  3. Relevant evaluation settings: The standard benchmarks are complemented by cross-lingual and non-lexical reference experiments.

Critical weaknesses
  1. The methodological novelty appears limited: The method combines established techniques and inherits transcript-free conditioning from IndexTTS2. The integration is effective, but the distinct methodological contribution could be clearer.

  2. The main text provides insufficient component-level ablations: The main text does not isolate the benefits of residual recovery, acoustic length estimation, or ReFlow. This makes it difficult to assess how much each component contributes to the final speed–quality trade-off.

  3. Baseline inference settings need clarification: Although timing is conducted on the same GPU, the inference frameworks and optimization settings are not specified. These details are particularly relevant for AR baselines, whose decoding speed can depend substantially on the inference implementation, and would help establish the comparability of the speed measurements.

Questions for authors
  1. Could the authors provide component ablations comparing discrete-only distillation with residual recovery, and the renderer before and after ReFlow at the same sampling budget?

  2. How does the acoustic length estimator compare with a fixed tokens-per-syllable baseline, particularly in content accuracy and adaptation to the reference speaking rate?

1 comment
AuthorsOpenReview ↗

Response to Reviewer JLKC

Thank you for the clear summary, and for recognizing the practical motivation, the favorable speed–quality trade-off, and the relevance of our evaluation settings. We address your concerns point-by-point below.


W1. Limited methodological novelty.

A: Thank you for recognizing the practical motivation. Our contribution is a systematic redesign of a strong AR system into a NAR one that is fast, transcript-free, and needs no user-specified duration. This required solving two problems that do not arise in the AR setting: fixing the output length without a reference transcript, which our training-free length estimator addresses, and driving the teacher's continuous-latent renderer from discrete NAR outputs, which residual recovery addresses (dropping it raises WER/CER by 18–50% relative). The result combines the strengths of both families:

  • over 10× faster generation than IndexTTS2 for utterances longer than 5 s;
  • transcript-free cloning that holds on cross-lingual references, where transcript-dependent systems fail even with ground-truth transcripts (global response Q2);
  • a low conversion cost: 2k hours of teacher-generated data, without the teacher's 55k-hour corpus.

W2. Insufficient component-level ablations in the main text.

A: Thank you for this suggestion. We ran all three ablations (global response Q1 and Q3) and have added them to Appendix H of the revised submission; we will move the key table into the main text in the final version.


W3. Inference settings of the baselines in the timing comparison are not specified.

A: Thank you for pointing this out. All baselines were timed with their official implementations and default settings on the same A100 GPU. For IndexTTS2, we used the speed-up options supported by its official implementation: KV caching (enabled by default), fp16, and num_beams = 1 instead of the default 3. These choices favor the teacher's speed. Even so, the AR decoder needs one forward pass per generated token, whereas our NAR decoder uses a fixed 12 forward passes regardless of output length.


Q1. Discrete-only vs. residual distillation, and the renderer before vs. after ReFlow at the same budget.

A: As described in our global response Q1:

  • Discrete-only vs. residual: WER/CER is 7.6 / 2.4 / 2.8 / 3.8 without the residual and 5.7 / 1.7 / 1.5 / 3.1 with it (LibriSpeech / SeedTTS-en / SeedTTS-zh / AISHELL-1).
  • ReFlow at equal steps: SS improves by +0.014 to +0.028 at 8 steps and by +0.037 to +0.067 at 4 steps, with similar WER/CER. With 8 steps, ReFlow matches the original renderer at 25 steps.

Q2. The length estimator vs. a fixed tokens-per-syllable baseline, in content accuracy and rate adaptation.

A: As shown in our global response Q3:

  • Content accuracy: against a fixed rate set from the training corpus, the estimator is better on three of four sets (e.g., 5.7 vs. 6.1 on LibriSpeech) and worse on AISHELL-1 (3.1 vs. 2.4). The gain is modest, so our main results do not depend on the exact length estimate.
  • Rate adaptation: we test whether a faster or slower reference speaker leads to faster or slower output. For each test item, we measure the speaking rate of the reference and of the output as seconds per syllable, using unmodified references. Across items, we regress the output rate on the reference rate in log scale; the slope then measures relative change: 1 means the output rate changes proportionally with the reference rate, and 0 means no dependence. Because a speaker's rate also varies between their own utterances, we include the ground-truth target speech of the same speaker as a natural reference point. Slope:
SpeechLibriSpeechSeedTTS-enSeedTTS-zhAISHELL-1
Ground-truth target0.370.410.590.55
Ours0.270.130.020.10

On LibriSpeech, our output follows about three quarters of the natural dependence (0.27 vs. 0.37), and it follows more weakly on SeedTTS-en. On Mandarin, it barely follows the reference rate, because most Mandarin references are slower than the estimator's upper bound (12 codes/syllable) and are capped there. The bound is needed because the NAR student fills an overly long target with repeated content (global response Q3).

We will present the estimator as a simple transcript-free length prior with partial pace transfer in the final version.

Reviewer_5BZAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Tacit-TTS, an efficient transcript-free zero-shot voice cloning system distilled from IndexTTS2. It aims to reduce the inference latency of autoregressive speech generation while preserving speaker similarity and synthesis quality. The paper introduces three main technical components:

  1. Masked Non-Autoregressive Text-to-Semantic Generation: The autoregressive Text-to-Semantic (T2S) module of IndexTTS2 is replaced with a non-autoregressive model based on the masked prediction architecture of MaskGCT. Knowledge distillation is used to learn both discrete semantic tokens and continuous latent representations.
  2. Training-Free Acoustic Length Estimation: Acoustic energy peaks in the reference audio are used to estimate the syllable count and speaking pace, which are combined with the syllable count of the target text to predict the generation length, without requiring reference transcripts or an additionally trained duration prediction model.
  3. Semantic-to-Mel ReFlow Distillation: The flow-matching renderer of IndexTTS2 is retained, while ReFlow distillation reduces the number of sampling steps required for generation, further improving inference efficiency.

The paper evaluates Tacit-TTS on four datasets: LibriSpeech test-clean, SeedTTS test-en, SeedTTS test-zh, and AISHELL-1 test. It further examines transcript-free voice cloning using reference speech from eight other languages, infant babble, and synthetic gibberish. Experimental results show that Tacit-TTS achieves competitive speaker similarity on English datasets and reduces T2S inference latency from 5274 ms to 199 ms. For utterances of approximately 5 seconds, the generation latency decreases from 6149 ms to 509 ms, while maintaining synthesis quality relatively close to IndexTTS2.

However, the paper exhibits limited methodological novelty, as Masked Prediction, Knowledge Distillation, and ReFlow are all established techniques rather than newly proposed methods. The ability to handle cross-lingual and non-lexical references is also largely inherited from IndexTTS2 rather than newly introduced. Additionally, the paper contains formatting issues: all table captions are placed below the tables, contrary to the ICLR 2027 template requirements, and the numerical results in Table 1 are reported with inconsistent decimal precision.

Critical strengths
  1. Substantial improvement in generation efficiency. Tacit-TTS replaces the 527M-parameter autoregressive T2S model with a 269M-parameter non-autoregressive model. In Table 4, T2S latency decreases from 5274 ms to 199 ms, while S2A latency decreases from 875 ms to 310 ms.
  2. Competitive synthesis quality despite substantial efficiency improvements. In Table 1, Tacit-TTS achieves speaker similarity scores of 0.875 and 0.849 on LibriSpeech test-clean and SeedTTS test-en, respectively, outperforming all listed non-teacher baselines. Despite the architectural simplification, the model preserves much of the teacher's voice cloning capability using approximately 2,000 hours of distillation data, without requiring access to the teacher's original large-scale training corpus.
  3. Preservation of transcript-free capabilities under challenging reference conditions. Tables 2 and 3 show that Tacit-TTS largely retains the ability of IndexTTS2 to handle cross-lingual and non-lexical references after replacing the autoregressive T2S module with a non-autoregressive model, while several transcript-dependent baselines exhibit substantial performance degradation or generation failures.
Critical weaknesses
  1. The contributions of individual components are insufficiently validated. The Masked Prediction architecture is primarily adopted from MaskGCT, while ReFlow is an existing technique for accelerating generation. Although the overall model achieves substantial efficiency improvements, the paper lacks independent ablations of Continuous Latent Residual Recovery and ReFlow Distillation. For example, Appendix D only compares different sampling steps without evaluating the effect of ReFlow at the same number of steps, making it difficult to isolate the contributions of individual components to generation quality and efficiency.
  2. The proposed Training-Free Acoustic Length Estimation relies heavily on heuristic assumptions, while its advantages and robustness are insufficiently validated. The method estimates speaking pace using acoustic energy peaks, a fixed correction factor, and clipping thresholds, but its robustness under noisy or reverberant reference conditions remains unexplored. Furthermore, the method assumes that the target speech follows the average speaking pace of the reference audio, which may not hold when the target text differs in phonetic content, prosodic structure, or speaking style. Although Appendix F reports a duration correlation of 0.948, comparable to MaskGCT's 0.949, the paper lacks comparisons with simpler fixed-rate baselines and controlled evaluations of speaking-pace preservation.
  3. The cross-lingual and non-lexical reference evaluations provide limited evidence of new capabilities introduced by Tacit-TTS. The transcript-free conditioning mechanism is inherited from IndexTTS2, which already supports these scenarios. As shown in Tables 2 and 3, Tacit-TTS largely preserves the teacher model's existing capabilities rather than consistently improving upon them. Therefore, these experiments primarily demonstrate the preservation of transcript-free voice cloning after replacing the autoregressive T2S module, rather than establishing improved generalization attributable to the proposed method.
  4. The paper lacks subjective evaluation. Perceptual quality is primarily assessed using UTMOS and DNSMOS, without human listening tests.
  5. The paper does not fully comply with the ICLR 2027 formatting requirements. The official template specifies that table captions should appear above the tables, whereas all table captions in this paper are placed below them. Additionally, Table 1 reports numerical results with inconsistent decimal precision (e.g., WER values of 5.999 and 5.54).
Questions for authors
  1. Why does the paper report 2,544 samples for LibriSpeech test-clean, whereas the standard dataset and IndexTTS2 evaluation contain 2,620 samples? Were the baseline results evaluated on the same subset as Tacit-TTS?
  2. In Section 3.1.1, the correction factor κ is set to 0.92 for Mandarin targets and 1 otherwise, while the syllable counts for mixed-language text are obtained by summing the estimates from different languages. How is κ determined for mixed Mandarin–English inputs? Is a single correction factor applied to the entire text, or are language-specific factors applied separately?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 5BZA

Thank you for the thorough and constructive review, and for recognizing our efficiency gain, competitive synthesis quality, and preserved transcript-free capability. We address your concerns point-by-point below.

W1. Missing ablations of residual recovery and of ReFlow at the same number of steps.

A: Thank you for this suggestion. As described in our global response Q1, we ran both ablations:

  • Residual recovery: disabling it raises WER/CER on all four test sets (e.g., LibriSpeech 5.7 → 7.6), with either renderer.
  • ReFlow: at the same number of steps, it improves SS by +0.014 to +0.028 at 8 steps and by +0.037 to +0.067 at 4 steps, with similar WER/CER.

We have added both ablations to Appendix H of the revised submission. We agree that masked prediction and ReFlow are established techniques. Our contribution is converting a strong transcript-free AR system into an efficient NAR one with only 2k hours of teacher-generated data, and the residual decomposition is what makes this conversion work.

W2. The length estimator lacks robustness tests, a fixed-rate baseline, and a pace-preservation evaluation.

A: Following your suggestion, we conducted three experiments.

(1) Robustness. We added white noise or reverberation to the references. Duration MAE (s) / Pearson r of the length estimate:

Reference conditionLibriSpeechSeedTTS-enSeedTTS-zhAISHELL-1
Clean0.78 / 0.9800.54 / 0.8751.06 / 0.6091.18 / 0.811
White noise, SNR 10 dB0.83 / 0.9750.54 / 0.8680.83 / 0.5630.73 / 0.792
Reverberation, RT60 0.6 s0.83 / 0.9760.51 / 0.8781.08 / 0.6041.10 / 0.805

The length estimate does not degrade: MAE rises by at most 0.05 s.

(2) Fixed-rate baseline. As shown in our global response Q3, the estimator's gain over a fixed rate is modest, so our main results do not depend on the exact length estimate. The upper clamp, however, is necessary: removing it raises WER/CER on all four sets.

(3) Pace preservation. On unmodified references, we measured how the output pace follows the reference pace, as a log-scale slope (1 = proportional, 0 = no dependence). The same speaker's ground-truth target shows the natural level:

SlopeLibriSpeechSeedTTS-enSeedTTS-zhAISHELL-1
Ground-truth target0.370.410.590.55
Ours0.270.130.020.10

Pace transfers partially on English, but not on Mandarin, where most references reach the upper clamp. A controlled time-stretch test gives a weaker response. We will revise the pace-transfer claim and list it as a limitation in the final version, together with prosody and style differences between reference and target, which the estimator does not model.

W3. The cross-lingual and non-lexical abilities are inherited from IndexTTS2.

A: We agree. These experiments show that the capability is preserved after replacing the AR T2S with a NAR model. This was not guaranteed. Unlike the AR teacher, which generates until a stop token, a NAR model must fix the output length in advance, and existing NAR systems use the reference transcript for this. Our length estimator must therefore work on such references without a transcript. As shown in our global response Q2, transcript-dependent baselines do not recover on FLEURS even with ground-truth transcripts. We will rephrase Sections 4.3 and 4.4 to emphasize preservation rather than improvement in the final version.

W4. No subjective evaluation.

A: We agree this is important. As described in our global response Q4, we could not run a properly sized listening study within the rebuttal period and will include one in the final version. In the meantime, we have uploaded audio samples of all six systems to the supplementary material.

W5. Caption placement and decimal precision.

A: Thank you for pointing this out. In the revised submission, all captions are above the tables, and Table 1 consistently uses three decimals.

Q1. LibriSpeech test set size mismatch (2,544 vs. 2,620).

A: Thank you for pointing this out. A bug in our data loader caused 76 utterances to be missing from the evaluation. We have fixed the bug and updated the results in the manuscript. On the full 2,620 utterances, Tacit-TTS achieves 5.562% WER and 0.873 SS, with minimal impact on the results and no change in the ranking in Table 1. The baseline results are taken from the IndexTTS2 paper, and we follow the same evaluation settings.


Q2. How is κ\kappa determined for mixed Mandarin–English text?

A: A single factor is applied to the whole sentence: κ=0.92\kappa = 0.92 whenever the target contains any Chinese character. Its value was set during development to correct an overcount of about 12% by the acoustic detector on SeedTTS-zh references, i.e., with the help of test references. However, as shown in our global response Q3, setting κ=1\kappa = 1 changes CER by at most 0.1, because most Mandarin references reach the upper clamp either way. We will clarify the mixed-language rule and the choice of κ\kappa in the final version.

Reviewer_FEy7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Tacit-TTS, a transcript-free zero-shot TTS system distilled from IndexTTS2. It replaces the teacher’s autoregressive text-to-semantic decoder with iterative masked prediction, estimates target length from acoustic syllable peaks in the reference and syllable counts in the target text, and predicts the teacher’s continuous residual latent in addition to discrete semantic tokens. The inherited flow-matching renderer is adapted and ReFlow-distilled from roughly 25 to 4–8 steps. Experiments report competitive English/Chinese synthesis, improved robustness to cross-lingual and non-lexical references, and substantially faster generation.

Critical strengths
  1. The paper addresses a practically meaningful combination of requirements: transcript-free conditioning, non-autoregressive semantic generation, and efficient acoustic rendering. The integration of discrete-code distillation with continuous latent recovery is technically coherent.

  2. The efficiency gain is substantial and mostly localized by stage: Table 4 supports a 26.5× T2S latency reduction and a 12.1× reduction for T2S+S2A generation relative to the teacher. The evaluation also covers unusually challenging cross-lingual and non-lexical references.

Critical weaknesses
  1. The cross-lingual and non-lexical experiments do not isolate the effect of transcript dependence. FLEURS provides reference transcripts, but no oracle-transcript control is reported; failures therefore conflate ASR errors, unsupported prompt languages, and differences among complete TTS systems. Moreover, the synthetic-gibberish references have exact generating strings in Table B.1 and were synthesized by IndexTTS2 using its official voices, favoring the teacher/student family while other systems receive noisy ASR output.
  2. The paper lacks ablations for its central mechanisms: residual-latent recovery, reference-derived length estimation, renderer adaptation, and ReFlow are never isolated. The pooled duration correlation in Figure F.1 is also insufficient to validate reference-pace transfer because it can be dominated by target-text length and cannot reveal systematic duration bias. Finally, the claim that 8→12 T2S steps “consistently” improves content accuracy conflicts with Table D.1, where SeedTTS-English WER worsens for every reported S2Mel setting.
Questions for authors
  1. Does the proposed estimator transfer reference speaking pace under controlled same-text/different-pace references? Please report duration bias/MAE and per-dataset results rather than only pooled Pearson correlation.
1 comment
AuthorsOpenReview ↗

Response to Reviewer FEy7

Thank you for the precise and constructive review, and for recognizing the practical value of combining transcript-free conditioning with NAR generation and efficient rendering. We address your concerns point-by-point below.


W1. The cross-lingual and non-lexical experiments do not isolate transcript dependence.

A: Thank you for the clear experimental suggestion.

  • FLEURS: we reran all four transcript-dependent baselines with ground-truth reference transcripts and correct language labels (global response Q2). Ground-truth transcripts do not help on average and hurt badly for some languages (e.g., CosyVoice2 → English: 46.0 → 63.4 WER), so the cross-lingual failures are not caused by ASR errors. We have added this experiment as Appendix I and corrected the related claims in the abstract and introduction.
  • Synthetic gibberish: we agree that this set favors the teacher family. Since the speech is non-lexical, a ground-truth transcript is not well defined. We have labeled the set as teacher-generated in Section 4.1, and in the final version we will base the non-lexical claim primarily on the real infant-babble set, for which no transcript exists.

W2.1. Missing component ablations.

A: Thank you for this suggestion. As described in our global response Q1 and Q3, we ablated each component with everything else held fixed:

  • Residual recovery: disabling it raises WER/CER on all four sets, with either renderer. We disable it at inference time, which keeps the discrete codes identical, so the change can be attributed directly to the residual.
  • ReFlow: at equal sampling steps, it improves SS by +0.014 to +0.028 at 8 steps and by +0.037 to +0.067 at 4 steps, with similar WER/CER.
  • Length estimation: compared with a fixed pace, the ground-truth length, and no clamp, the upper clamp is essential, and the gain over a fixed pace is modest.

We have added these ablations to Appendix H.


W2.2. The pooled duration correlation (Figure F.1) can be dominated by text length and hides systematic bias.

A: We agree. Per-dataset duration bias (s) / MAE (s) / Pearson r:

Target lengthLibriSpeechSeedTTS-enSeedTTS-zhAISHELL-1
Fixed pace−0.30 / 0.86 / 0.975−0.15 / 0.52 / 0.873−1.27 / 1.28 / 0.597−1.35 / 1.35 / 0.799
Ours−0.30 / 0.78 / 0.980−0.26 / 0.54 / 0.875−0.99 / 1.06 / 0.609−1.17 / 1.18 / 0.811

As you anticipated, this view reveals a systematic negative bias of about 1 s on Mandarin, with two roughly equal sources: (1) the upper clamp, since Mandarin test speech is slower than the bound and than our Mandarin training data; and (2) the generated speech being about 0.5 s shorter than the allotted length, which the teacher also shows (−0.4 to −0.5 s). Removing the clamp removes most of the bias but raises CER substantially (global response Q3).


W2.3. The claim that 8→12 T2S steps "consistently" improves content accuracy conflicts with Table D.1.

A: Thank you for catching this; you are correct. We have revised the sentence in Section 4.5 to "improves content accuracy on LibriSpeech and both Mandarin sets (slightly worse on SeedTTS-en)."


Q1. Does the estimator transfer reference pace under controlled same-text / different-pace references?

A: We ran two tests. In both, we report the slope of log output pace against log reference pace (1 = proportional, 0 = no dependence). Per-dataset bias and MAE are given in W2.2.

(1) Controlled test. We time-stretched 10 references to 0.8×, 1.0×, and 1.25× with a phase vocoder and used the same 5 targets. The slope is 0.05 for our system on both LibriSpeech and AISHELL-1, vs. 0.62 / 0.76 for IndexTTS2 and 0.43 / 0.51 for our estimate before clamping. The response is weak because most references already reach the upper bound, and because the syllable detector merges adjacent syllables on sped-up audio.

(2) Natural references. On unmodified references, we compare with the ground-truth target of the same speaker, which shows how strongly pace naturally carries over:

SpeechLibriSpeechSeedTTS-enSeedTTS-zhAISHELL-1
Ground-truth target0.370.410.590.55
IndexTTS20.440.470.640.57
Ours0.270.130.020.10

Our system carries over about three quarters of the natural dependence on LibriSpeech, less on SeedTTS-en, and almost none on Mandarin, where the clamp fixes the pace. In the final version, we will revise the pace-transfer claim accordingly and list pace transfer for slow references as a limitation.

Reviewer_GZPaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Tacit-TTS, which is a transcript-free zero-shot voice cloning system distilled from IndexTTS2. It replaces autoregressive generation with masked non-autoregressive generation, estimates speech length in a training-free way, and speeds up flow-matching using ReFlow distillation. Evaluations on English and Mandarin datasets show competitive voice cloning quality with over 10× faster generation than IndexTTS2 for speech longer than 5 seconds.

Critical strengths
  1. Tacit-tts significantly speeds up synthetic audio generation: Table 4 shows significant, well-established gain in speed. Though, Fig 3 honestly shows the speedup shrinking beyond over a minute.
  2. Tacit-tts establishes data-efficient distillation from IndexTTS2: The residual decomposition lets a non-autoregressive student feed the teacher’s renderer without retraining it from scratch. It uses about 2k hours of teacher-generated data instead of the teacher’s 55k hours of data.
Critical weaknesses
  1. Limited perceptual validation: SS and predicted MOS sometimes exceed ground-truth scores, but without human similarity or preference tests, preservation of perceived speaker identity and naturalness remains insufficiently validated.
  2. Speed–accuracy trade-off: Reported WER/CER is mostly higher than the teacher, indicating a content-accuracy drop that should be explicitly weighed against the speed gains.
  3. Limited validation of length estimation: The correlation between generated and actual speech lengths does not, by itself, show that using the reference audio’s speaking pace helps. Comparing against a fixed pace and the true target length, alongside clamp and cross-language calibration ablations, would better isolate its contribution.
  4. Limited analysis beyond empirical tuning: Although step-count ablations support the chosen configuration, the paper does not investigate why distillation enables few-step generation or how ReFlow performance depends on its 8,000-pair training budget.
Questions for authors
  1. Could you report the semantic frame rate ff, the fraction of references reaching each clamp bound, and how κ\kappa, the bounds, and peak thresholds were chosen, including whether test data informed these choices; compare against fixed-pace and ground-truth-length baselines; and report per-dataset pace error (seconds per syllable) alongside duration correlation?
  2. Could you rerun the FLEURS baselines with gold reference transcripts and correct language labels where supported, to distinguish ASR-related failures from limitations of the synthesis models?
  3. Could you compare against a matched student without residual recovery (λ=0\lambda=0, with the residual output disabled and renderer conditioning clearly specified) to establish its contribution to speaker similarity and content accuracy?
  4. Appendix E mentions four ESD voices reserved for evaluation, but I could not find corresponding results; could you clarify their use and compare emotion-control performance with IndexTTS2 on held-out speakers, ideally including human judgments of emotion fidelity?
3 comments
AuthorsOpenReview ↗

Response to Reviewer GZPa

Thank you for the careful reading, and for recognizing the speed gain and the data-efficient distillation. We address your concerns point-by-point below.


W1. Limited perceptual validation without human listening tests.

A: We agree that automatic metrics cannot replace human listeners. SS above ground truth is not specific to our model; the teacher shows the same effect (Table 1). As described in our global response Q4, we will include a listening study in the final version, and we have uploaded audio samples of all six systems to the supplementary material.


W2. The speed–accuracy trade-off relative to the teacher should be made explicit.

A: We agree. Tacit-TTS trades a small loss in content accuracy for a large gain in speed: its WER/CER is 0.7–2.4 points higher than IndexTTS2, while its total latency is 977 ms instead of 6632 ms, about 6.8× faster (Table 4). We will state this trade-off explicitly in the final version.


W3. The length estimator is validated only by duration correlation; compare with a fixed pace and the true length, and ablate the clamp and calibration.

A: Following your suggestion, we ran these comparisons (global response Q3):

  1. The gain over a fixed pace is modest (better on three of four sets, worse on AISHELL-1), so our main results do not depend on the exact length estimate.
  2. The clamp is necessary: without it, WER/CER rises on all four sets (e.g., LibriSpeech 5.7 → 10.4). Even the ground-truth length is worse when the true pace is slower than the bound.
  3. Setting κ=1\kappa = 1 changes CER by at most 0.1.

We have added the fixed-pace, ground-truth-length, and no-clamp comparisons to Appendix H.


W4. Why distillation enables few-step generation, and how ReFlow depends on its 8,000-pair budget.

A: Few steps. From 8 to 4 steps, the original renderer loses 0.03–0.06 SS, while the ReFlow renderer loses only 0.01–0.02 (global response Q1). This is consistent with ReFlow straightening the transport paths, which reduces the discretization error of few-step sampling.

Coupling budget. We reran ReFlow from the original renderer with 1k, 2k, and 4k couplings, using the same recipe as the 8k model in the paper. Averaged over the four test sets:

RendererSS, 8 stepsSS, 4 stepsWER/CER (%), 8 steps
Original (no ReFlow)0.8120.7633.05
ReFlow, 1k couplings0.8300.8083.02
ReFlow, 2k couplings0.8320.8112.98
ReFlow, 4k couplings0.8340.8182.90
ReFlow, 8k couplings (paper)0.8350.8203.00

ReFlow is data-efficient: 1k couplings already recover most of the SS gain, and performance saturates around 4k. WER/CER does not depend on the budget.

Q1. Frame rate, clamp-hit fractions, how the constants were chosen, and per-dataset pace error.

A:

  • Frame rate: f=50f = 50 codes/s.
  • Clamp-hit fractions: the upper bound (12 codes/syllable) is reached by 56.5% / 41.9% / 90.5% / 95.0% of references on LibriSpeech / SeedTTS-en / SeedTTS-zh / AISHELL-1; the lower bound (7) is never reached.
  • Constants: κ\kappa and the upper bound were selected during development with the help of test-set references; the peak-detection thresholds (25 dB floor, 3 dB prominence, 100 ms gap) were set by hand. κ=0.92\kappa = 0.92 corrects a 12% syllable overcount of the acoustic detector on SeedTTS-zh references; as noted in W3, it has a negligible effect, and re-estimating it on AISHELL-1 training speakers gives κ≈1.00\kappa \approx 1.00. The upper bound is supported independently: for references slower than the bound, the ground-truth length raises WER/CER from 7.6 to 13.0 (LibriSpeech) and from 3.9 to 9.8 (AISHELL-1). We will state this in the final version.
  • Per-dataset error: see the table below. Mandarin shows a negative bias of about 1 s. About half comes from the clamp, because Mandarin test speech is slower than the bound; the other half comes from the generated speech being shorter than the allotted length, which the teacher also shows.

Pace error (ms/syllable, bias / MAE) and duration error (s, bias / MAE / Pearson r):

MetricTarget lengthLibriSpeechSeedTTS-enSeedTTS-zhAISHELL-1
PaceFixed pace−19 / 37−14 / 33−66 / 67−92 / 92
PaceOurs−22 / 35−20 / 35−53 / 56−81 / 81
DurationFixed pace−0.30 / 0.86 / 0.975−0.15 / 0.52 / 0.873−1.27 / 1.28 / 0.597−1.35 / 1.35 / 0.799
DurationOurs−0.30 / 0.78 / 0.980−0.26 / 0.54 / 0.875−0.99 / 1.06 / 0.609−1.17 / 1.18 / 0.811

Q2. Rerun the FLEURS baselines with ground-truth reference transcripts and correct language labels.

A: As shown in our global response Q2, ground-truth transcripts do not reduce the baselines' FLEURS errors on average and increase them sharply for some languages (e.g., CosyVoice2 → English: 46.0 → 63.4 WER). The failures therefore come from conditioning on text in languages the systems do not support well, not from ASR errors. We have added this experiment as Appendix I.


Q3. Compare with a student without residual recovery (λ=0\lambda = 0), with the renderer conditioning clearly specified.

A: We disabled the residual output (Δ=0\Delta = 0), so the renderer is conditioned on vq2emb(y)\mathrm{vq2emb}(y) only. WER/CER (%) / SS:

RendererResidualLibriSpeechSeedTTS-enSeedTTS-zhAISHELL-1
ReFlow (ours, 8 steps)off7.6 / 0.8732.4 / 0.8442.8 / 0.8163.8 / 0.785
ReFlow (ours, 8 steps)on5.7 / 0.8701.7 / 0.8411.5 / 0.8273.1 / 0.802
Original (25 steps)off8.4 / 0.8772.9 / 0.8483.0 / 0.8254.1 / 0.792
Original (25 steps)on5.6 / 0.8751.7 / 0.8471.5 / 0.8333.0 / 0.804

The residual lowers WER/CER on all four sets with either renderer. We disable it at inference time rather than retrain a student, because this keeps the discrete codes identical and isolates the effect of the residual. These results are in Appendix H; we are happy to add a retrained λ=0\lambda = 0 student in the final version if you consider it necessary.


Q4. The four ESD voices reserved for evaluation, and emotion control compared with IndexTTS2.

A: We apologize for the confusion. ESD has 20 speakers: we trained on 16 and held out 4 (2 English, 2 Mandarin). On these held-out speakers, we compared emotion control with IndexTTS2 using the same references (2 per speaker and emotion, 5 emotions) and targets. Emotion accuracy is the fraction of outputs that emotion2vec classifies as the reference emotion (references: 95% in English, 100% in Mandarin); ES is the emotion2vec similarity used in the IndexTTS2 paper.

SystemLanguageEmotion accuracyESSSWER/CER (%)
IndexTTS2English78.3%0.8040.8320.00
Tacit-TTSEnglish66.7%0.7740.8140.92
IndexTTS2Mandarin91.7%0.9180.8590.50
Tacit-TTSMandarin81.7%0.8480.7850.98

Tacit-TTS largely preserves the teacher's emotion control (ES 0.03–0.07 lower); for reference, the IndexTTS2 paper reports ES of 0.66–0.89 for prior systems on its own test set. We will add a human evaluation of emotion fidelity in the final version.

Thank you for the detailed rebuttal and for conducting the additional experiments. I appreciate the effort to address the concerns raised during the review. In particular, the gold-transcript control (Q2) provides convincing evidence that the cross-lingual baseline failures are not simply due to ASR errors, while the coupling-budget sweep (W4) offers useful evidence of ReFlow's data efficiency. I also appreciate the authors' transparency regarding test-set tuning.

However, after considering the responses and additional results, several of my main concerns remain unresolved:

  1. Perceptual validation: The authors have kept perceptual evaluation for the camera-ready version.

  2. Length estimation: The upper length clamp is active for approximately 42–95% of the references, raising concerns about the reliability of the proposed length estimation approach. Moreover, fixed pacing performs similarly within the margin of noise on three datasets and even outperforms the proposed approach on AISHELL-1. The fact that the constants were tuned using test references further weakens the evaluation.

  3. Residual ablation: Disabling Δ only at inference time is not an adequate substitute for training a separate student model with λ = 0. This setup may overestimate the actual contribution of the residual component. Additionally, the discrete-only variant matches or outperforms SS on both English datasets, which does not fully support the explanation provided in Section 3.3.

  4. Emotion control: The approximately 10-percentage-point drop in emotion accuracy on the small held-out samples is not negligible. Therefore, describing emotion control as "largely preserved" seems overly optimistic.

  5. Accuracy and efficiency claims: The reported WER/CER values are approximately 70–80% higher than those of the teacher model on two benchmarks, which raises concerns about the claimed performance quality. Furthermore, the abstract's claim of achieving ">10× beyond 5 s" does not appear consistent with Table 4, where the reported speedup is approximately 6.8×. This discrepancy should be clarified.

Overall, I appreciate the authors' efforts and the additional evidence provided in the rebuttal. However, the responses do not sufficiently resolve my central concerns regarding perceptual quality, evaluation reliability, ablation validity, and the strength of the performance claims.

Therefore, I retain my original rating.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response

We sincerely thank all reviewers for their time and their thoughtful, constructive feedback. We are encouraged that the reviewers found:

  • the efficiency gain substantial and well localized by stage.
  • the residual decomposition an effective way to distill from the teacher with a small data budget.
  • the cross-lingual and non-lexical evaluations challenging and relevant.

Below, we address the concerns shared by several reviewers. Specific questions are answered in the individual replies. All new results will be included in the final version.


Experimental setup for the new results. All new experiments use a fixed evaluation subset of 100 utterances per test set, stratified by target length. Every configuration uses the same items and seeds, which gives 400 utterances per configuration. On this subset, our default system obtains 5.7 / 1.7 / 1.5 / 3.1 WER/CER (%), close to its full-set results in Table 1.


Q1. Component-level ablations

A: We thank the reviewers for this suggestion, which makes the contribution of each component much clearer. We ablated each component with everything else kept fixed.

Residual recovery. "Discrete only" disables the residual (Δ = 0), so the renderer is conditioned on vq2emb(y) alone. WER/CER (%) / SS:

ConfigLibriSpeechSeedTTS-enSeedTTS-zhAISHELL-1
Discrete only, ReFlow renderer (8 steps)7.6 / 0.8732.4 / 0.8442.8 / 0.8163.8 / 0.785
+ residual, ReFlow renderer (8 steps), ours5.7 / 0.8701.7 / 0.8411.5 / 0.8273.1 / 0.802
Discrete only, original renderer (25 steps)8.4 / 0.8772.9 / 0.8483.0 / 0.8254.1 / 0.792
+ residual, original renderer (25 steps)5.6 / 0.8751.7 / 0.8471.5 / 0.8333.0 / 0.804

The residual lowers content error on all four test sets, by 18–50% relative, and improves Mandarin similarity. The gain holds with the original IndexTTS2 renderer as well, so it does not depend on ReFlow.

ReFlow, compared with the original IndexTTS2 renderer at the same number of sampling steps. WER/CER (%) / SS:

RendererStepsLibriSpeechSeedTTS-enSeedTTS-zhAISHELL-1
Original255.6 / 0.8751.7 / 0.8471.5 / 0.8333.0 / 0.804
Original85.7 / 0.8561.8 / 0.8171.5 / 0.7993.2 / 0.775
ReFlow (ours)85.7 / 0.8701.7 / 0.8411.5 / 0.8273.1 / 0.802
Original45.5 / 0.8222.1 / 0.7721.6 / 0.7393.2 / 0.720
ReFlow (ours)45.6 / 0.8591.6 / 0.8311.5 / 0.8043.4 / 0.787

At equal steps, ReFlow mainly preserves speaker similarity: +0.014 to +0.028 SS at 8 steps, and +0.037 to +0.067 at 4 steps. Content error is essentially unchanged. With 8 steps, ReFlow matches the original 25-step renderer (SS gap ≤ 0.006) at about one third of the S2A cost.

The length-estimator ablations are reported in Q3 below.


Q2. Ground-truth-transcript control on FLEURS for transcript-dependent baselines

A: On FLEURS, we reran the four transcript-dependent baselines with the ground-truth reference transcript (and the correct language label where supported), keeping everything else unchanged. WER/CER (%), ASR transcript → ground-truth transcript, averaged over 8 languages × 5 targets:

Reference → targetF5-TTSMaskGCTCosyVoice2SparkTTSIndexTTS2Ours
FLEURS → en33.7 → 50.611.3 → 30.5 *46.0 → 63.450.9 → 66.1 *0.00.3
FLEURS → zh30.1 → 29.822.8 → 36.8 *66.7 → 92.243.3 → 57.1 *1.31.4

* Averaged over successful items. MaskGCT first converts the reference transcript into phonemes (pronunciation symbols) with a language-specific front end, which does not cover Arabic, Hindi, or Russian, so it cannot synthesize from these 3 of the 8 FLEURS languages. SparkTTS produces empty or garbled audio on 20–25% of items. The averages for these two systems therefore cover fewer languages.

With ground-truth transcripts, the baselines do not improve on average, and some languages degrade sharply: on Yoruba references, for example, CosyVoice2's WER on English targets rises from 2% to 118%. ASR errors are therefore not what causes the cross-lingual failures. Instead, these systems struggle to condition on text in languages they do not support well, a limitation that transcript-free cloning avoids by design.

Q3. Experiments on the training-free length estimator

A: WER/CER (%) with different ways of setting the target length:

Target lengthLibriSpeechSeedTTS-enSeedTTS-zhAISHELL-1
Fixed pace (training-corpus rate)6.11.91.62.4
Ground-truth duration8.02.42.17.5
Ours without clamp10.42.02.78.3
Ours5.71.71.53.1
  1. The upper clamp is necessary. Removing it raises WER/CER on all four sets. Even the ground-truth length is worse: when asked to speak slower than the bound, the NAR student fills the extra frames with drawn-out or repeated content.
  2. The gain over a fixed pace is modest. The estimator is better on three of four sets and worse on AISHELL-1, so our main results do not depend on the exact length estimate. We will state this accurately and present the estimator as a simple length prior that needs neither a transcript nor a trained duration model.
  3. Pace transfer is partial. On English, the output partially follows the reference pace; on Mandarin, most references reach the clamp, so the pace is fixed. We will revise the pace-transfer claim and list it as a limitation (details in our reply to FEy7).
  4. Constants. The Mandarin calibration κ\kappa and the upper bound were set during development with the help of test-set references. We chose κ=0.92\kappa = 0.92 because, on SeedTTS-zh references, the acoustic detector counted about 12% more syllables than the reference transcripts. Its effect on the results is negligible: setting κ=1\kappa = 1 changes CER by at most 0.1, and re-estimating it on training speakers gives κ≈1.00\kappa \approx 1.00. The upper bound is supported independently by point 1. We will clarify this in the final version.

Q4. Lack of human evaluation

A: We agree that automatic metrics cannot replace human listeners. Due to the limited rebuttal period, we could not conduct a properly sized listening study; we will include blinded CMOS and SMOS studies in the final version. In the meantime, we have uploaded audio samples to the Supplementary Material (open index.html in the zip). They compare all six systems side by side, with the same reference and target text, on LibriSpeech, AISHELL-1, and cross-lingual FLEURS references.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.