- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces TGSLM, a speech-to-speech pretraining method that creates an internal text interface for knowledge transfer while preserving speech-only input and output. The proposed method predicts text before generating its corresponding speech segments, allowing knowledge distillation from a pretrained text LM by matching next-token prediction distributions.
Critical strengths
The paper addresses an important challenge in speech-to-speech language modeling by introducing an internal text interface that enables knowledge transfer from pretrained text LMs while maintaining speech-only input and output.
Critical weaknesses
-
The novelty is limited. Previous studies, such as TWIST, have already explored transferring pretrained text LM capabilities to speech language models, while BLSP-KD, DiVA, and SALAD have investigated knowledge distillation from text LMs to speech-input, text-output models. Although this paper extends knowledge distillation to speech-to-speech models through internal text predictions, the underlying distillation strategy largely follows existing techniques. Therefore, the methodological novelty beyond adapting existing knowledge transfer approaches to a speech-to-speech framework remains unclear.
-
The paper focuses on evaluating pretrained models without investigating their performance after task-specific fine-tuning. Since TGSLM is proposed as a speech-to-speech pretraining method, evaluating whether its improvements transfer to downstream tasks would strengthen the paper. In particular, fine-tuning TGSLM for speech-based instruction following and evaluating it on benchmarks such as VoiceBench would help demonstrate the practical benefits of the proposed pretraining approach.
-
The pretraining experiments rely primarily on Emilia and Emilia-YODAS, without examining the generalizability of the proposed method across different training corpora. Moreover, the evaluation mainly focuses on generative perplexity, StoryCloze, and Known-1000. Additional evaluations on established speech language modeling benchmarks, such as sWUGGY and sBLIMP, as well as few-shot ASR on LibriSpeech would provide a more comprehensive assessment of the proposed pretraining method.
-
It does not evaluate whether the generated speech accurately reflects the internally predicted text. Since TGSLM generates text before its corresponding speech segments, measuring WER between the predicted text and the transcription of the generated speech would provide a direct assessment of text-speech consistency.
-
The comparisons are primarily focused on general speech language modeling performance. In particular, the paper lacks direct comparisons with alternative knowledge distillation methods and relevant text-pretrained speech models under comparable training conditions. Stronger baselines would help clarify whether the observed improvements arise from the proposed approach rather than existing text-guided modeling and knowledge transfer techniques.
-
The performance improvement from knowledge distillation alone is relatively modest (58.9 -> 56.9). Moreover, these improvements come with reduced speaker similarity (0.42 -> 0.36) and UTMOS-v2 (2.56 -> 2.41). A substantial gap in factual knowledge also remains between TGSLM-distill and its pretrained text LM backbone on Known-1000, although the evaluation modalities differ. These results raise questions about the effectiveness of knowledge transfer and the trade-off between linguistic and acoustic performance.
Questions for authors
Please see the concerns detailed in the Weaknesses.
We thank the reviewer for the feedback. We address the main concerns below.
Novelty [Critical Weakness 1]. Our goal is not to introduce a new distillation algorithm, but to demonstrate that a simple approach can effectively retain pretrained text LM knowledge during speech-to-speech pretraining, which to our knowledge has not been shown before. Our framework provides an internal text interface that enables knowledge transfer while maintaining speech-only input and output.
Downstream evaluation [Critical Weakness 2]. Our focus is general-purpose speech LM pretraining rather than task-specific post-training. For this reason, we evaluate open-ended speech continuation, factual knowledge, and StoryCloze, leaving downstream instruction following for future work.
Benchmark coverage [Critical Weakness 3]. We consider sWUGGY and sBLIMP narrower benchmarks, as they evaluate specific linguistic phenomena using synthetic speech derived from a single text source. In contrast, our open-ended generation evaluation uses natural speech from Emilia and assesses broader linguistic quality. We believe this better reflects the capabilities targeted by general-purpose generative speech pretraining. We would be happy to add sWUGGY and sBLIMP, but we would appreciate your thoughts on what this would add beyond the evaluations we have already done.
Text-speech consistency [Critical Weakness 4]. We measured WER between internally generated text and ASR-transcribed speech on the Emilia validation set: