- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates how effectively synthetic speech can replace real speech during SpeechLM pretraining. The authors find that synthetic-only training increases joint pretraining loss, while mixtures containing up to 50% synthetic speech show no detectable degradation. Increasing speaker diversity and channel augmentation reduce the gap.
Critical strengths
- The rigour in experimental setup, such as using same sentences for real and synthetic speech, and have identical token budgets, model sizes for the comparisons is commendable.
- The paper is well written, with clearly motivated hypotheses that are systematically investigated and addressed.
Critical weaknesses
My biggest concern is the lack of rigorous quality control over the generated synthetic speech. The authors do not adequately characterize synthesis errors, and their filtering experiments are limited to removing the lowest-ranked 20% using relatively simple metrics. Without evaluating a strictly cleaned synthetic corpus (e.g., using CER thresholds), it is difficult to determine whether the reported synthetic–real gap reflects inherent limitations of synthetic speech or simply avoidable generation errors.
This is particularly concerning because the real-data baseline includes carefully curated datasets such as Granary, which employ extensive data cleaning and filtering. Applying substantially less rigorous quality control to synthetic speech potentially makes the comparison unfair. The large differences observed across languages further reinforce this concern.
Also, joint loss may not always reflect speech understanding performance. For example, synthetic-only Spanish training increases joint loss by 2.3% but improves transcript-token accuracy by 6.1 percentage points (Table 10).
Questions for authors
- What are the CER/WER distributions of the generated speech across languages? How does the synthetic-real gap change under stricter quality filtering?
- What explains the substantial gap between Spanish and Korean? How much can be attributed to TTS quality, speaker coverage, or tokenizer behavior?
- Do the main conclusions hold when evaluating speech loss, text loss, and transcription accuracy separately instead of joint loss?
2 comments
Follow-up on Our Rebuttal Before the Discussion Deadline
Dear Reviewer,
As the discussion period ends in less than 24 hours, we kindly wanted to follow up on our rebuttal. We would greatly appreciate it if you could review our responses and share any remaining concerns before the deadline. Thank you for your time and consideration.
We thank the reviewer for recognizing the rigor of our controlled setup. We added stricter quality-control experiments and a per-metric analysis to address the concerns raised.
We examined clip-level character error rate (CER) distributions and trained models with much stricter filtering.
First, standard quality metrics do not suggest that synthetic speech is systematically worse than real speech. In every language, synthetic speech has both (i) fewer clips above 10% CER and (ii) a smaller 90th percentile of clip-level CER when compared to real speech (Appendix B.2, Table 7). Quality scores like UTMOS, NISQA, DNSMOS, and SNR also point in the same direction (Appendix B.2, Tables 6–7).
More directly, we tested stricter filtering: CER ≤5%, retaining 60% of tokens based on CER, multiple quality scores, similarity to real speech, and acoustic-feature coverage. All are compared with size-matched random subsets. None of the 13 comparisons improves joint loss significantly over random selection, and five are worse by 0.4–1.4%. These results do not support audio quality, as quantified by standard quality markers, as the main explanation for the real-synthetic gap. Clip-level CER distributions are added in Appendix B.2 (Table 7), and the stricter filtering methods and their confidence intervals are in Appendix B.5 (Table 8).
We cannot identify a single factor that fully explains the larger Korean gap. We investigated several possible contributors by estimating how separate diagnostic comparisons reduce Korean's excess gap relative to the mean of the four European languages (1.17B):
Speaker coverage is the largest factor in our analysis. However, Korean still has a much larger synthetic-only joint-loss gap than Spanish (20.3% vs. 6.4%) when both are limited to ~100 reference speakers. The Korean–Spanish difference also persists under a different TTS system (CosyVoice 3: 36.3% vs. 1.1%) and a different speech tokenizer (Mimi: 18.7% vs. 1.5%). Thus, while limited speaker diversity, acoustic coverage, and tokenizer degradation appear to contribute, we cannot isolate a dominant cause of the Korean–Spanish difference. The reference-speaker controls are in Appendix B.8 (Table 13), the acoustic and tokenizer diagnostics in Appendix B.13 (Figure 13), and the alternative TTS/tokenizer experiments in Appendices C.1–C.2.
The submission already reports the three metrics separately: speech and text loss in Section 4.1, all three for every language and both model sizes in Appendix B.7 (Table 11), and transcript accuracy for each experiment in Appendix B.10.
We also checked each claim across 14 settings (six languages plus multilingual, and two model sizes). While synthetic-only training worsens joint and speech loss across all 14 configurations, statistically significant degradations in text loss and transcript accuracy occur in only 5/14. We additionally examined whether the following two main findings hold under these separate evaluations: (1) the synthetic-only gap grows with training budget, and (2) ending training on real speech performs better than ending on synthetic speech. Both findings hold across all 14 settings for joint and speech loss, but only 9/14 for text loss and 8/14 for transcript accuracy. The revised manuscript now distinguishes the conclusions across metrics in Appendix B.7 (Table 11) and Appendix B.10 (Figure 12), and limits claims about speech understanding accordingly.