acceptodds
Under review as a conference paper at ICLR 2027

How Far Can Synthetic Speech Take Us? Synthetic Data Recipes for Pretraining Multilingual Speech Language Models

Abstract

Scaling pretraining data has been central to the progress in large language models (LLMs). However, extending scaling recipes to speech-enabled multimodal LLMs is constrained by the scarcity of large, diverse, and permissively licensed speech corpora, a bottleneck even more acute in the speech domain for underrepresented languages. Synthetic speech data can potentially address this gap, but its effectiveness as a training substitute for real data remains unquantified. In this work, we systematically study this question across six languages, varying real-synthetic mixtures, training curricula, and augmentations to the synthetic corpus. Our results show that the quality gap between real and synthetic-only pretraining increases as the training-token budget increases. However, this gap largely disappears when models are pretrained on joint mixtures of synthetic and real speech data: up to 50% of the training data can be synthetic with no detectable degradation across any tested language, while some languages tolerate substantially larger synthetic fractions. Diversifying synthetic data by broadening speaker coverage and performing simple augmentations like reverberation and noise also narrows the synthetic-real gap. Overall, our work provides practical guidance for constructing and using synthetic speech to scale multilingual speech language models under real-world data availability and licensing constraints.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Audio Language Modelling, Synthetic Data, Multimodal, Large Language Models, Speech Language Modelling, Multimodal LLMs, Multilingual

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RUFmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates how effectively synthetic speech can replace real speech during SpeechLM pretraining. The authors find that synthetic-only training increases joint pretraining loss, while mixtures containing up to 50% synthetic speech show no detectable degradation. Increasing speaker diversity and channel augmentation reduce the gap.

Critical strengths
  • The rigour in experimental setup, such as using same sentences for real and synthetic speech, and have identical token budgets, model sizes for the comparisons is commendable.
  • The paper is well written, with clearly motivated hypotheses that are systematically investigated and addressed.
Critical weaknesses

My biggest concern is the lack of rigorous quality control over the generated synthetic speech. The authors do not adequately characterize synthesis errors, and their filtering experiments are limited to removing the lowest-ranked 20% using relatively simple metrics. Without evaluating a strictly cleaned synthetic corpus (e.g., using CER thresholds), it is difficult to determine whether the reported synthetic–real gap reflects inherent limitations of synthetic speech or simply avoidable generation errors.

This is particularly concerning because the real-data baseline includes carefully curated datasets such as Granary, which employ extensive data cleaning and filtering. Applying substantially less rigorous quality control to synthetic speech potentially makes the comparison unfair. The large differences observed across languages further reinforce this concern.

Also, joint loss may not always reflect speech understanding performance. For example, synthetic-only Spanish training increases joint loss by 2.3% but improves transcript-token accuracy by 6.1 percentage points (Table 10).

Questions for authors
  1. What are the CER/WER distributions of the generated speech across languages? How does the synthetic-real gap change under stricter quality filtering?
  2. What explains the substantial gap between Spanish and Korean? How much can be attributed to TTS quality, speaker coverage, or tokenizer behavior?
  3. Do the main conclusions hold when evaluating speech loss, text loss, and transcription accuracy separately instead of joint loss?
2 comments

We thank the reviewer for recognizing the rigor of our controlled setup. We added stricter quality-control experiments and a per-metric analysis to address the concerns raised.

W1/Q1/W2. Quality control of synthetic speech: It is difficult to determine whether the reported synthetic–real gap reflects inherent limitations of synthetic speech or simply avoidable generation errors

We examined clip-level character error rate (CER) distributions and trained models with much stricter filtering.

First, standard quality metrics do not suggest that synthetic speech is systematically worse than real speech. In every language, synthetic speech has both (i) fewer clips above 10% CER and (ii) a smaller 90th percentile of clip-level CER when compared to real speech (Appendix B.2, Table 7). Quality scores like UTMOS, NISQA, DNSMOS, and SNR also point in the same direction (Appendix B.2, Tables 6–7).

More directly, we tested stricter filtering: CER ≤5%, retaining 60% of tokens based on CER, multiple quality scores, similarity to real speech, and acoustic-feature coverage. All are compared with size-matched random subsets. None of the 13 comparisons improves joint loss significantly over random selection, and five are worse by 0.4–1.4%. These results do not support audio quality, as quantified by standard quality markers, as the main explanation for the real-synthetic gap. Clip-level CER distributions are added in Appendix B.2 (Table 7), and the stricter filtering methods and their confidence intervals are in Appendix B.5 (Table 8).

Q2. Spanish vs. Korean gap: What explains the substantial gap between Spanish and Korean?

We cannot identify a single factor that fully explains the larger Korean gap. We investigated several possible contributors by estimating how separate diagnostic comparisons reduce Korean's excess gap relative to the mean of the four European languages (1.17B):

Factor examinedReduction in excess joint gapReduction in excess text gap
Matched reference-speaker pool30%37%
Channel augmentation15%15%
Speech-tokenizer degradation2%16%
Read speech only (no conversational speech)1%0%

Speaker coverage is the largest factor in our analysis. However, Korean still has a much larger synthetic-only joint-loss gap than Spanish (20.3% vs. 6.4%) when both are limited to ~100 reference speakers. The Korean–Spanish difference also persists under a different TTS system (CosyVoice 3: 36.3% vs. 1.1%) and a different speech tokenizer (Mimi: 18.7% vs. 1.5%). Thus, while limited speaker diversity, acoustic coverage, and tokenizer degradation appear to contribute, we cannot isolate a dominant cause of the Korean–Spanish difference. The reference-speaker controls are in Appendix B.8 (Table 13), the acoustic and tokenizer diagnostics in Appendix B.13 (Figure 13), and the alternative TTS/tokenizer experiments in Appendices C.1–C.2.

W3/Q3. Separate speech/text/transcript metrics: Do the main conclusions hold when evaluating speech loss, text loss, and transcription accuracy separately instead of joint loss?

The submission already reports the three metrics separately: speech and text loss in Section 4.1, all three for every language and both model sizes in Appendix B.7 (Table 11), and transcript accuracy for each experiment in Appendix B.10.

We also checked each claim across 14 settings (six languages plus multilingual, and two model sizes). While synthetic-only training worsens joint and speech loss across all 14 configurations, statistically significant degradations in text loss and transcript accuracy occur in only 5/14. We additionally examined whether the following two main findings hold under these separate evaluations: (1) the synthetic-only gap grows with training budget, and (2) ending training on real speech performs better than ending on synthetic speech. Both findings hold across all 14 settings for joint and speech loss, but only 9/14 for text loss and 8/14 for transcript accuracy. The revised manuscript now distinguishes the conclusions across metrics in Appendix B.7 (Table 11) and Appendix B.10 (Figure 12), and limits claims about speech understanding accordingly.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Before the Discussion Deadline

Dear Reviewer,

As the discussion period ends in less than 24 hours, we kindly wanted to follow up on our rebuttal. We would greatly appreciate it if you could review our responses and share any remaining concerns before the deadline. Thank you for your time and consideration.

Reviewer_ubb6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents an empirical study on using TTS-generated synthetic speech to pretrain multilingual speech-text language models (490M and 1.17B parameters) across six languages (EN, DE, ES, FR, ZH, KO). Using Qwen3-TTS to synthesize audio from real transcripts, the authors evaluate different real-synthetic mixture ratios, training curricula, quality filtering, speaker diversity, and acoustic augmentations under fixed token budgets (primarily 2B tokens). They find that while synthetic-only training lags behind real-only training, mixing up to 50% synthetic data or ending training on real data closes most of the loss gap, and channel/speaker diversity matters more than quality filtering.

Critical strengths
  1. Important and timely research question with large-scale empirical effort: Investigating how far synthetic speech can substitute for real speech in multimodal SpeechLM pretraining is a highly practical direction, given licensing and data bottlenecks in scaling audio models. The controlled setup (matching transcripts across real and synthetic splits) and the exploration of curricula and augmentations offer useful references for practitioners.
Critical weaknesses
  1. Cross-language differences are heavily confounded by Qwen3-TTS's language-specific capability and reference speaker pool sizes (Section 4.1, Table 5, Table 6): The paper repeatedly emphasizes that the synthetic-real gap varies drastically across languages (e.g., 1.0% for Spanish vs. 16.7%–20.3% for Korean and 8.1%–11.0% for Chinese). However, all main experiments rely on a single TTS model (Qwen3-TTS), whose synthesis quality and zero-shot cloning capability are inherently uneven across languages. For instance, in Table 6, Qwen3-TTS has a 29.8% Whisper CER on Chinese—twice as bad as real Chinese data (14.7%). Even worse, in Table 5, the Common Voice reference speaker pool used for synthesis has 16k–83k speakers for European languages, 5,104 for Chinese, and only 107 speakers for Korean. Since Section 4.3 proves that low speaker diversity severely hurts loss, attributing the large gap in Korean/Chinese to "language differences" without controlling for TTS quality and speaker pool size is flawed.
  2. Missing "50% Real-Only" baseline to validate the 50% mixture claim (Section 4.2, Figure 4): A central claim of the paper is that up to 50% of the training data can be synthetic (i.e., 1B real + 1B synthetic tokens) with no detectable degradation compared to 2B real-only training. However, the paper never shows how much performance drops if we simply train on only the 50% real data (either stopping at 1B real tokens, or repeating that 50% real subset for 2 epochs to match the 2B token budget). Looking at the validation curves in Figure 6, the loss plateau between 50% (1B) and 100% (2B) is quite flat. If training on 50% real data alone (1B or repeated to 2B) already achieves nearly the same loss as 100% real data, then the added 50% synthetic data is merely "not hurting" rather than actively contributing.
Questions for authors

Clarification Requests:

  1. Comparison against 50% Real-Only (1B tokens or 2-epoch repeated 2B tokens): In Figure 4 (at the 2B budget), when 50% real data is mixed with 50% synthetic data, how does it compare to (a) training on that exact 50% real data alone (1B tokens), and (b) repeating that 50% real data twice to reach the same 2B compute budget? Does the 50% synthetic data provide actual positive gains over simply reusing the remaining 50% real data?
  2. Disentangling language effects from Qwen3-TTS quality and speaker count:
    • In Table 5, Korean only uses 107 reference speakers while English uses 83,057 and Spanish/French/German use 16k–19k. What is the cross-language synthetic-real gap if you fix the reference speaker pool to 100 speakers across all six languages?
    • Since Qwen3-TTS has a much higher CER in Chinese (29.8% vs. 14.7% real in Table 6) and Appendix C.1 only tests a second TTS (ZipVoice-Distill) on EN and ZH, how can we tell whether the cross-language ordering is a property of the languages rather than an artifact of Qwen3-TTS's language-specific training data?
2 comments

We thank the reviewer for recognizing the importance of the problem and the value of our controlled experimental setup. We address the two concerns below.

W1/Q2. Cross-language differences are heavily confounded by Qwen3-TTS's language-specific capability and reference speaker pool sizes

We added experiments with 100 reference speakers per language for Spanish and German (in addition to the already existing English, French, and Chinese in Figure 5), two additional TTS systems, and a second speech tokenizer, and find that the broader language ordering holds across all of these choices.

First, we repeated synthetic-only pretraining with 100 reference speakers for each of the six languages (We don't do a separate run for Korean, which has 107 reference speakers). The results below show the relative joint-loss increase over real-only training (1.17B, 2B tokens; lower is better):

LanguageFull pool100 speakers
Korean+20.3%+20.3%
Chinese+11.0%+15.6%
German+5.7%+10.8%
English+4.9%+9.0%
French+4.6%+9.0%
Spanish+2.2%+6.4%

The six languages retain the same order of synthetic-real gaps as with their full pools, even though reducing the pool worsens the gap in all of them. The complete six-language comparison is in Appendix B.8 (Table 13).

Second, the order of the gaps is also consistent beyond Qwen3-TTS and WavTokenizer (Appendices C.1–C.2). The following tables show that CosyVoice 3 and the Mimi tokenizer give the same order: Korean > Chinese > German > Spanish order, and ZipVoice-Distill gives Chinese > English (ZipVoice only supports these two languages).

Synthesis / tokenizerModelKOESZHDEEN
Qwen3-TTS / WavTokenizer1.17B+20.3+2.2+11.0+5.7+4.9
ZipVoice / WavTokenizer1.17B——+13.9—+7.6
CosyVoice 3 / WavTokenizer1.17B+36.3+1.1+15.7+4.3—
Qwen3-TTS / WavTokenizer490M+18.8+2.4+10.6+4.9+5.1
Qwen3-TTS / Mimi490M+17.2+2.0+8.7+5.1—

Another thing to note is that the language-level WER and speaker similarity reported by Qwen3-TTS do not correlate with our synthetic–real gaps (Spearman ρ = −0.09 for WER and +0.09 for speaker similarity).

We also investigated the high Chinese Whisper CER noted by the reviewer. Just 1.9% of the synthetic generations, mostly hallucinated and excessively long, account for 67% of the 29.8% CER in Table 6. We remove these samples from the training clips using a simple heuristic of filtering any generated audio longer than 60 seconds. Consequently, the training clips' CER is 16.1%, which is very close to real speech’s 14.7%.

Therefore, the cross-language differences observed in our work cannot all be attributed to reference speaker pool size or TTS transcription errors and the language-level synthesis quality of Qwen3-TTS. We have revised Appendix B.2 with these details.

W2/Q1. Real-only 50% baselines: Does the 50% synthetic data provide actual positive gains over simply reusing the remaining 50% real data?

To test this, we took half of the real data used in the 2B 50/50 mixture and use the following data setups to train two additional 1.17B models: (a) 1B real tokens, and (b) the same 1B real tokens repeated twice. The following results are joint-loss increases relative to the non-repeated 2B real-only baseline:

Training data (joint-loss change vs. full real-only, 2B)KoreanSpanish
Real half only, 1B tokens+3.5%+12.3%
Real half repeated, 2B tokens+0.5%+0.4%
50/50 real–synthetic mixture, 2B tokens+0.6%−1.9%
Both Korean and Spanish columns show that 1B real-only tokens are substantially worse than the non-repeated 2B tokens. Repeating the available real data recovers most of this loss gap. However, in Spanish, replacing the repeated real speech with synthetic speech produces a further 2.3% reduction in joint loss. Even though, for Korean, the synthetic replacement of the repeated real speech for Korean is much closer in terms of joint-loss increase; these results suggest that synthetic speech is at least as good as the repeated real data, and unlike real data whose repetition value can diminish eventually [1, 2], can add more value when synthetic speech is generated from new text-only content.

We also test the effect of synthetic speech generated from new text content - FineWeb, by replacing the second pass over 1B real tokens with synthetic speech of new FineWeb text. The joint-loss for this model does not differ significantly (+0.3%) but this lowers the text loss by 13.5% on test words that occur only in the FineWeb text but not in the real transcripts. The repetition controls and FineWeb experiment are now detailed in Appendix B.11 (Table 15).

[1] Khandelwal et al., arXiv 2026: Scaling Laws for Multimodal Data Mixtures. [2] Muennighoff et al., NeurIPS 2023: Scaling Data-Constrained Language Models

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Before the Discussion Deadline

Dear Reviewer,

As the discussion period ends in less than 24 hours, we kindly wanted to follow up on our rebuttal. We would greatly appreciate it if you could review our responses and share any remaining concerns before the deadline. Thank you for your time and consideration.

Reviewer_699POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a careful and systematic empirical study of how far synthetic speech can substitute for real speech when pretraining multilingual speech language models. Using matched transcripts to control linguistic content, the authors quantify the real–synthetic performance gap across six languages and two model sizes under token-matched training budgets, and then analyze how training curricula, mixture ratios, and synthetic data construction (speaker diversity, acoustic/channel augmentation, quality filtering) affect this gap. The key findings are that (a) synthetic-only pretraining underperforms real-only and the gap widens with training budget, (b) mixing synthetic with real data or ending training on real data closes most of this gap, with up to 50% synthetic tokens showing no detectable degradation at 2B tokens across all languages, and (c) diversity-oriented strategies (broader speaker coverage and channel augmentation) are more effective than standard quality filtering at improving the utility of synthetic speech.

Critical strengths
  • A well-controlled matched-transcript framework isolates acoustic effects from linguistic content, which is rarely done at this scale across multiple languages.
  • The work operationalizes the “synthetic–real gap” and examines its scaling with token budget, providing new insights about how synthetic data utility changes as pretraining grows.
  • Multiple languages (EN/DE/FR/ES/KO/ZH), two model sizes, and token-matched training make comparisons fair and interpretable.
  • Realistic training setups (Megatron-LM, WavTokenizer, 4k context) and uncertainty estimates via paired bootstrap plus seed variance provide credible statistical support.
  • The problem is well-motivated, the methodology is clearly described, and figures concisely communicate the main effects with confidence intervals.
Critical weaknesses
  • Reliance on a single speech tokenizer (WavTokenizer at 40 tps) and a single TTS system (Qwen3-TTS) may limit generality; it is unclear how conclusions transfer across tokenizers, token rates, and other TTS systems or synthesis paradigms.
  • The main claims are evaluated predominantly via pretraining loss (joint/speech/text), with limited visibility into downstream task performance in the main paper.
  • The “50% synthetic without degradation” finding is firmly shown at 2B tokens; it is unclear whether this threshold shifts at larger budgets where the synthetic-only gap grows.
  • While the matched-transcript design isolates acoustics, it does not explore scenarios where synthetic speech also introduces novel linguistic content (a major practical use case).
  • Quality-filtering negative results are interesting but could be underpowered; only a few criteria (ASR error, transcript expansion, VAD ratio) are tested and on a fixed removal rate (20%).
  • The causes of large text-loss degradation for ZH/KO under synthetic-only training are speculated but not deeply probed (e.g., phonology/orthography, TTS pronunciation fidelity, tokenizer–grapheme alignment).
  • The study could better connect to recent large-scale discrete audio modeling and scaling-law work (e.g., SODA) that also use validation NLL as a reliable proxy, reinforcing the choice of loss as a primary metric and clarifying differences in tokenization and interleaving strategies.
Questions for authors
  • Can you summarize key downstream task results (e.g., ASR WER, spoken QA) corresponding to the main conditions (synthetic-only vs mixed vs real-only; with/without channel augmentation) in the main text to corroborate the loss-based conclusions?
  • How sensitive are your conclusions to the choice of tokenizer and token rate (e.g., different RVQ codecs, 20 vs 40 vs 100 tps)? Do you expect the “gap grows with budget” trend and the “50% synthetic at 2B” rule-of-thumb to hold across tokenizations?
  • Do you have TTS-side diagnostics per language (ASR WER on synthesized speech, UTMOS/NISQA, phoneme error rates) and correlations with the observed gaps, particularly for Chinese and Korean where text loss degrades most?
  • In the quality-filtering experiments, did you try dynamic thresholds or multi-objective selection (e.g., Pareto over MER/UTMOS/variance) and per-language tuning? Could smarter selection close more of the gap than random downsampling?
  • How does the 50% synthetic threshold change at larger budgets (e.g., 4B, 6B tokens) where the synthetic-only gap increases? Any evidence that the tolerable synthetic fraction shrinks with scale?
  • Did you explore more targeted or realistic channel simulation (e.g., high-fidelity RIRs, measured codecs) akin to 2608.20971, and if so, does higher-fidelity channel modeling bring additional gains beyond your current recipe?
  • Could you release the augmentation and mixture curricula code and a small synthetic/real split to facilitate reproducibility and follow-up studies?
3 comments
AuthorsOpenReview ↗

Response Part 1/2: Generality, downstream evidence, longer training, new linguistic content

We thank the reviewer for recognizing the controlled design and breadth of the evaluation. We address the concerns in two parts.

W1/Q2. Generality: Reliance on a single speech tokenizer … and a single TTS system … may limit generality

We extended our evaluation to two more TTS systems and a second speech tokenizer (also operating at different tokens/sec). The following table shows the relative joint-loss changes for synthetic-only training relative to real-only training with an identical experimental setup (e.g., model size, 2B tokens, and tokenizer); positive is worse:

Synthesis / tokenizerModelKOESZHDEEN
Qwen3-TTS / WavTokenizer1.17B+20.3+2.2+11.0+5.7+4.9
ZipVoice / WavTokenizer1.17B——+13.9—+7.6
CosyVoice 3 / WavTokenizer1.17B+36.3+1.1+15.7+4.3—
Qwen3-TTS / WavTokenizer490M+18.8+2.4+10.6+4.9+5.1
Qwen3-TTS / Mimi490M+17.2+2.0+8.7+5.1—

The relative ordering of languages on synthetic-real gap is the same as under Qwen3-TTS: Korean > Chinese > German > Spanish gap. Moreover, 50% real / 50% synthetic mixture shows no significant joint-loss increase over real-only training either with ZipVoice-Distill or with Mimi (all four languages). Appendix C.1 (Table 16) and C.2 (Table 17) contain the detailed results.

W2/Q1. downstream evidence: limited visibility into downstream task performance in the main paper

We have surfaced the transcript-accuracy findings in the main text and expanded the analysis in Appendix B.10 (Figure 12). Appendix B.7 (Table 11) reports speech loss, text loss, and transcription metrics separately.

W3/Q5. Longer training budgets: it is unclear whether this threshold shifts at larger budgets where the synthetic-only gap grows

We have extended the 1.17B synthetic-real proportion experiments from Section 4.2 to 4B tokens for Spanish and Korean, the languages with the smallest and largest synthetic-only gaps. Even at 4B tokens, at 87.5% synthetic proportion in Spanish has a lower joint loss than real-only training. However, for Korean, the real-synthetic joint loss gap at 50% synthetic rises from +0.60% (not significant) to +0.73% (significant), shifting the undetectable proportion of synthetic data to 37.5%. We have updated the pdf with the detailed results in Appendix B.12.

W4. New linguistic content: it does not explore scenarios where synthetic speech also introduces novel linguistic content

We tested this for English (1.17B, 2B tokens). Our baseline is a model trained on 1B real-speech tokens, repeated twice for a total of 2B training tokens. We then train a model with 1B real-speech tokens and 1B tokens of Qwen3-TTS speech synthesized from previously unseen FineWeb text. On words that are only present in the FineWeb speech and not in the real speech, the second model has a 13.5% lower text loss and a higher transcription accuracy (4.1% vs 2.6%) when compared to the baseline. Text loss is also 10.4% lower than for a model trained on the baseline data plus the same FineWeb sentences as text (2.09B tokens). Although overall joint loss does not improve significantly, these gains on newly covered words demonstrate a potential benefit of generating speech from text outside the available real-speech transcripts. Refer to Appendix B.11 in the revised pdf for more details.

AuthorsOpenReview ↗

Response Part 2/2: Filtering, gap diagnosis, and practical details

W5/Q4. Filtering: Quality-filtering negative results are interesting but could be underpowered

We extended the submission's original 20% filtering ablation to stricter thresholds and more elaborate selections. The following table reports relative joint-loss increases of each filtering criterion with respect to a random selection of the same number of tokens. The values in the brackets are 95% CIs (negative = the selection is better):

SelectionZHKOES
CER ≤ 5%−0.11 [−0.40, +0.18]—−0.13 [−0.44, +0.17]
Lowest CER (retain 60%)−0.15 [−0.44, +0.14]+0.62 [+0.33, +0.90]+0.92 [+0.61, +1.22]
Pareto over five scores (60%)+0.37 [+0.08, +0.66]+1.43 [+1.14, +1.71]+0.56 [+0.26, +0.86]
Most real-like, classifier (60%)−0.22 [−0.51, +0.07]−0.13 [−0.41, +0.16]—
Match real-speech features (60%)−0.24 [−0.53, +0.05]+0.22 [−0.07, +0.50]−0.23 [−0.53, +0.07]

Pareto ranks clips by Pareto dominance over CER, phone error, UTMOS, voice similarity, and duration; "most real-like" is determined by a logistic-regression-based classifier that predicts if a specific clip is real; and ‘match real-speech features’ checks if a synthetic clip falls within the real speech’s distribution of pitch, noise, reverberation, speaking rate, and speaker. We observe that even for the stronger filtering criteria, no selection is significantly better than random. Appendix B.5 contains the detailed results.

W6/Q3. Gap diagnosis: The causes of large text-loss degradation for ZH/KO … are speculated but not deeply probed

We investigate several possible contributors, but cannot identify a single factor that fully explains the larger Chinese/Korean gap. The following table contains the features studied, and how much they reduce Korean’s excess gap over the mean of the four European languages:

Factor examinedReduction in excess joint gapReduction in excess text gap
Matched reference-speaker pool30%37%
Channel augmentation15%15%
Speech-tokenizer degradation2%16%
Read speech only (no conversational speech)1%0%

Speaker coverage is the largest factor in our analysis. However, when comparing to Spanish which has the smallest real-synthetic gap in our analysis, Korean still has a much larger synthetic-only joint-loss gap (20.3% vs. 6.4%) even when both are limited to ~100 reference speakers. The Korean–Spanish difference also persists under a different TTS system (CosyVoice 3: 36.3% vs. 1.1%) and a different speech tokenizer (Mimi: 17.2% vs. 2.0%).

Additionally, for Chinese, we try isolating the text-loss gap on only ‘read’ speech (sentences read aloud from a script, not spoken spontaneously in conversation) and the gap turned out to be 57% lower than for the full evaluation set. This suggests that our synthesis pipeline may not reproduce the diversity of the Chinese evaluation set, which includes read, spontaneous, and meeting speech kinds.

Finally, the language-level WER and speaker similarity reported by Qwen3-TTS do not correlate with our synthetic–real gaps either (Spearman ρ = −0.09 for WER and +0.09 for speaker similarity).

Thus, while limited speaker diversity, acoustic coverage, and tokenizer degradation appear to contribute, we cannot isolate a dominant cause of the degradation in Chinese/Korean. The reference-speaker controls are in Appendix B.8 (Table 13), the acoustic and tokenizer diagnostics in Appendix B.13 (Figure 13), and the alternative TTS/tokenizer experiments in Appendices C.1–C.2.

Q6. Better channel augmentation: Did you explore more targeted or realistic channel simulation

Our recipe already uses measured room impulse responses (RIRS_NOISES), recorded noise (MUSAN), and codec effects (MP3, GSM, band-limiting), and these improve joint loss over unaugmented synthetic speech in all 14 evaluated settings. More elaborate channel simulation is a promising direction, and we leave this for future work. The augmentation recipe is specified in Appendix A.3 (Table 4), with per-metric results in Appendix B.6 (Table 10).

W7. Practical details: The study could better connect to recent large-scale discrete audio modeling … SODA)

We have acknowledged SODA and related scaling studies in the Related Work in the revised pdf.

Q7. Practical details: Could you release the augmentation and mixture curricula code…follow-up studies?

We plan to release the code, training configurations, and paired real/synthetic sets where the source licenses permit redistribution upon acceptance.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Before the Discussion Deadline

Dear Reviewer,

As the discussion period ends in less than 24 hours, we kindly wanted to follow up on our rebuttal. We would greatly appreciate it if you could review our responses and share any remaining concerns before the deadline. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.