acceptodds
Under review as a conference paper at ICLR 2027

Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions

Abstract

Text language models (LMs) acquire broad linguistic and factual capabilities through large-scale pretraining, while comparable capabilities remain difficult to learn from speech alone. Recent work has shown that knowledge distillation from pretrained text LMs offers a promising way to transfer their linguistic and factual capabilities to speech models. However, existing distillation approaches have largely focused on speech-input, text-output models. This idea has yet to be explored for speech-to-speech models, which operate on both speech input and speech output. In this work we introduce a speech-to-speech pretraining method that creates an internal text interface for knowledge transfer while preserving the speech-only input and output interface. The model first consumes speech context without its transcription, switches to text prediction, and then uses the predicted text to lead future speech generation word by word. It also provides an interface for knowledge distillation: the internal text predictions can be directly distilled from the base text LM. To the best of our knowledge, this is the first application of text LM distillation to a speech-to-speech LM. Under matched training conditions, the proposed sequence construction substantially improves the generative perplexity of spoken continuations over speech-only and interleaved speech-text models, approaching the performance of a text-leading inner-monologue model with external ASR prompt transcription. More importantly, distillation improves factual performance under both text and speech evaluation, indicating better retention of knowledge from the base text language model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech LM, distillation, pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_k7q4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TGSLM, a speech-to-speech pretraining method that creates an internal text interface for knowledge transfer while preserving speech-only input and output. The proposed method predicts text before generating its corresponding speech segments, allowing knowledge distillation from a pretrained text LM by matching next-token prediction distributions.

Critical strengths

The paper addresses an important challenge in speech-to-speech language modeling by introducing an internal text interface that enables knowledge transfer from pretrained text LMs while maintaining speech-only input and output.

Critical weaknesses
  1. The novelty is limited. Previous studies, such as TWIST, have already explored transferring pretrained text LM capabilities to speech language models, while BLSP-KD, DiVA, and SALAD have investigated knowledge distillation from text LMs to speech-input, text-output models. Although this paper extends knowledge distillation to speech-to-speech models through internal text predictions, the underlying distillation strategy largely follows existing techniques. Therefore, the methodological novelty beyond adapting existing knowledge transfer approaches to a speech-to-speech framework remains unclear.

  2. The paper focuses on evaluating pretrained models without investigating their performance after task-specific fine-tuning. Since TGSLM is proposed as a speech-to-speech pretraining method, evaluating whether its improvements transfer to downstream tasks would strengthen the paper. In particular, fine-tuning TGSLM for speech-based instruction following and evaluating it on benchmarks such as VoiceBench would help demonstrate the practical benefits of the proposed pretraining approach.

  3. The pretraining experiments rely primarily on Emilia and Emilia-YODAS, without examining the generalizability of the proposed method across different training corpora. Moreover, the evaluation mainly focuses on generative perplexity, StoryCloze, and Known-1000. Additional evaluations on established speech language modeling benchmarks, such as sWUGGY and sBLIMP, as well as few-shot ASR on LibriSpeech would provide a more comprehensive assessment of the proposed pretraining method.

  4. It does not evaluate whether the generated speech accurately reflects the internally predicted text. Since TGSLM generates text before its corresponding speech segments, measuring WER between the predicted text and the transcription of the generated speech would provide a direct assessment of text-speech consistency.

  5. The comparisons are primarily focused on general speech language modeling performance. In particular, the paper lacks direct comparisons with alternative knowledge distillation methods and relevant text-pretrained speech models under comparable training conditions. Stronger baselines would help clarify whether the observed improvements arise from the proposed approach rather than existing text-guided modeling and knowledge transfer techniques.

  6. The performance improvement from knowledge distillation alone is relatively modest (58.9 -> 56.9). Moreover, these improvements come with reduced speaker similarity (0.42 -> 0.36) and UTMOS-v2 (2.56 -> 2.41). A substantial gap in factual knowledge also remains between TGSLM-distill and its pretrained text LM backbone on Known-1000, although the evaluation modalities differ. These results raise questions about the effectiveness of knowledge transfer and the trade-off between linguistic and acoustic performance.

Questions for authors

Please see the concerns detailed in the Weaknesses.

1 comment

We thank the reviewer for the feedback. We address the main concerns below.

  1. Novelty [Critical Weakness 1]. Our goal is not to introduce a new distillation algorithm, but to demonstrate that a simple approach can effectively retain pretrained text LM knowledge during speech-to-speech pretraining, which to our knowledge has not been shown before. Our framework provides an internal text interface that enables knowledge transfer while maintaining speech-only input and output.

  2. Downstream evaluation [Critical Weakness 2]. Our focus is general-purpose speech LM pretraining rather than task-specific post-training. For this reason, we evaluate open-ended speech continuation, factual knowledge, and StoryCloze, leaving downstream instruction following for future work.

  3. Benchmark coverage [Critical Weakness 3]. We consider sWUGGY and sBLIMP narrower benchmarks, as they evaluate specific linguistic phenomena using synthetic speech derived from a single text source. In contrast, our open-ended generation evaluation uses natural speech from Emilia and assesses broader linguistic quality. We believe this better reflects the capabilities targeted by general-purpose generative speech pretraining. We would be happy to add sWUGGY and sBLIMP, but we would appreciate your thoughts on what this would add beyond the evaluations we have already done.

  4. Text-speech consistency [Critical Weakness 4]. We measured WER between internally generated text and ASR-transcribed speech on the Emilia validation set:

Distillation weight λ00.10.250.50.75
WER (%)8.19.512.423.339.0
Reviewer_PtedOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TGSLM, an speech-to-speech generative language modeling approach that, as many previous works, generates speech hierarchically: first generating linguistic content as text, and then generating speech conditioned on it, word by word. Following previous works on text-speech cross-modal distillation, the authors additionally investigate providing additional supervision from a frozen text language model, a variant they call TGSLM-distill. They evaluate TGSLM models with a set of generation- and likelihood-based metrics measuring linguistic, acoustic and basic paralinguistic (speaker and emotion agreement) properties. TGSLM models outperform speech only and interleaved text-speech language models in linguistic metrics, and better preserve text-domain capabilities, while performing competitively on acoustic and paralinguistic metrics. TGSLM-distill shows the benefits of text-speech cross-modal distillation in linguistic tasks performance.

Critical strengths

While most of the stated contributions of the paper are, in my opinion, not meaningfully novel (see weaknesses section below), the evaluation of preservation of paralinguistic sensitivity provides some preliminary insight into a relatively underexplored question: whether cross-modal distillation from text LMs harms a speech model's ability to preserve paralinguistic information during speech generation. However, the evaluation is limited, particularly for emotion preservation, which is assessed on a small synthetic dataset using a single classifier. I therefore consider this a minor contribution.

Critical weaknesses

This work is largely incremental and contributes little novel knowledge beyond the strength stated above, corresponding to the third main contribution stated by the authors (L111). Below I address the other two state contributions and explain why, in my opinion, they should not be considered meaningfully novel:

  • “(...) we predict future text before its corresponding speech, while the input remains speech-only at inference” (L100): Hierarchical speech generation—generating first linguistic content as text and then generate speech conditioned on it—has been a well-established, and arguably the dominant, approach in the field for more than a year, as shown by many works like Moshi (Défossez et al., 2024), Mini-Omni (Xie and Wu, 2024), LLaMA-Omni (Fang et al., 2025), Qwen2.5-Omni (Xu et al., 2025), Kimi-Audio (2025) and many others. While the authors seem to distinguish their approach from other works like Qwen2.5-Omni by the fact they use a separate talker module for streaming speech generation, it should be noted that this is because the approach by the authors is considered to be largely inefficient due to its serial nature. Most models have opted for a much more efficient streaming or parallel generation than TGSLM's serial approach. This added inefficiency shouldn’t be a claim for novelty.

    Furthermore, the claim that text-leading approaches require a transcription of the speech prompt at inference time (L71–75 and Table 1) is misleading. Several of the aforementioned models already perform text-guided speech generation directly from speech inputs, without requiring a prompt transcript. Thus, the ability to generate text-guided speech without an input transcript is not a novel contribution of TGSLM.

    The authors also attempt to distinguish their work by emphasizing speech pretraining rather than dialogue post-training (L149–151), but this distinction is questionable. Moshi already employs its text-guided inner-monologue formulation during large-scale speech pretraining. More generally, applying an established modeling approach to a different training stage or data distribution does not, by itself, constitute a meaningful contribution.

  • “We distill a pretrained text LM into TGSLM’s intermediate text predictions (...) this is the first application of distillation from text LMs to speech-to-speech LMs” (L106). While indeed most of the previous works doing text-speech cross-modal distillation focused on speech-in-text-out models, the proposed distillation approach does not introduce any methodological contribution. Previous works like SALAD have already demonstrated the benefits of cross-modal text-speech distillation both for preserving the text-domain capabilities of the pretrained text LLM and for transferring these capabilities to text generation conditioned on speech context. Importantly, the setup in the present paper is essentially the same with respect to the distillation objective: the model predicts content as text conditioned on a speech context, and its text predictions are distilled from a frozen text LM. Therefore, the reported benefits are largely a reproduction of already established work. Moreover, previous works explore more challenging broad-domain benchmarks (e.g. MMLU, ARC-Challenge) where the claim of capability transfer is more meaningful than the basic knowledge recall and common sense reasoning benchmarks here.

    The main difference is that TGSLM additionally generates speech conditioned on the predicted text, introducing a speech prediction loss term that is jointly optimized with the distillation and text prediction loss. This could potentially introduce interesting interactions between speech generation and cross-modal distillation, but these are not investigated. While the authors ablate distillation and text replay, the speech-generation term is always present, and its effect therefore never isolated. Thus, the experiments do not address whether or how the speech-generation objective affects the effectiveness of cross-modal text-speech distillation, and therefore do not contribute any meaningful insight beyond what is known from other works.

    The claim that this is the first instance of text-LM distillation applied to speech-to-speech models also does not hold. X-OPD (Cao et al., 2026) applies cross-modal distillation from a text-only teacher to Qwen3-Omni, a speech-to-speech model. Similarly, Qwen3.5-Omni (Qwen Team, 2026) applies cross-modal distillation to transfer stronger text-conditioned capabilities to speech-conditioned inference. While these works differ from the specific distillation objective used here in that they are on-policy—different from this work, which like BLSP-KD, DiVA or SALAD, is off-policy—they demonstrate that transferring text-domain capabilities through distillation in speech-to-speech architectures is not novel.

Questions for authors

Do you have any analysis of the interplay between speech generation and cross-modal text-speech distillation? I think there is a potentially meaningful contribution in addressing this matter, particularly the interaction between cross-modal distillation—which may encourage invariance to acoustic and paralinguistic information—and the model's ability to preserve paralinguistic information for speech generation. However, I would like to note upfront that given my assessment that most of the paper's contributions are not novel, I do not anticipate raising my score above a weak reject, and I believe the paper needs substantial additional work before being suitable for publication at ICLR.

2 comments

We thank the reviewer for the detailed feedback. We would like to clarify the contributions of our work.

  1. Novelty of text-guided speech generation [Critical Weakness 1]. We acknowledge prior text-guided speech generation models, including Moshi, FLM-Audio, and Qwen2.5-Omni, as discussed in Sec. 2.2. However, these systems primarily investigate speech dialogue or instruction-following settings, whereas our work isolates text-guided generation in general-purpose speech LM pretraining. One important distinction from Thinker-Talker architectures, such as Qwen2.5-Omni, is that TGSLM jointly models text and speech in a single autoregressive sequence. In typical Thinker-Talker architectures, the Thinker does not condition on the audio generated by the Talker when predicting subsequent text. In contrast, TGSLM conditions subsequent text predictions on previously generated speech, allowing the model to incorporate its own acoustic realizations into future linguistic predictions. To our knowledge, prior work has not provided a direct, controlled comparison of speech-only, interleaved, and text-guided formulations for generative speech pretraining. Existing text-guided models are generally trained at substantially larger scales and often target spoken dialogue or instruction following. In contrast, we investigate the effectiveness of text guidance for open-ended speech continuation under comparable training settings.

  2. Cross-modal distillation in speech-to-speech models [Critical Weakness 2]. We agree that cross-modal distillation has been studied, but all listed prior work, including BLSP-KD, DiVA, SALAD, and X-OPD, focuses on speech-input, text-output generation. X-OPD's distillation experiments operate on text responses rather than generated speech. Qwen3.5-Omni supports speech generation, but its cross-modal distillation is applied to the Thinker during post-training, with a separately trained Talker (no dependency in the generated speech tokens). In contrast, we study distillation during generative speech-to-speech pretraining, jointly optimizing internal text predictions and speech generation.

  3. Interaction between distillation and speech generation [Critical Weakness 2; Question for Authors]. Yes, we have already investigated how distillation affects speech generation, including acoustic quality, speaker similarity, and emotion sensitivity. In Table 2, adding distillation does not further reduce speaker similarity (0.36 → 0.36) or UTMOS-v2 (2.41 → 2.41), suggesting that distillation and speech generation can be jointly optimized without additional degradation in these acoustic metrics. We additionally analyzed WER between internally generated text and ASR-transcribed speech on the Emilia validation set. Stronger distillation improves factual knowledge retention (Fig. 2), but increases text-speech disagreement. Together, these analyses characterize the effects of distillation on both acoustic properties and speech realization consistency.

Distillation weight λ00.10.250.50.75
WER (%)8.19.512.423.339.0

If this does not address your question, we would appreciate if you could clarify what you intended. Regardless of this paper’s fate at ICLR, we would be interested to know about suggested experiments that we may have missed.

Reviewer_PtedOpenReview ↗

Response to authors

Thank you for your response. Here some remarks:

On the clarification of novelty:

  • While you acknowledge prior work, you also present as the first of your main contributions what is to practical effects the same contribution that models like Moshi made over two years ago with their inner monologue. Conceptually, I see your proposed architecture roughly as that of pretrained Moshi with a single flattened output stream. I do not think you are positioning your work correctly within the existing literature.
  • I don't think that differentiating from other models doing speech generation with intermediate text generation on the basis that they target dialogue and instruction-following usage instead of pretraining is convincing, considering that these models underwent pretraining before being trained for dialogue or instruction-following. In the case of Moshi—the most relevant comparsion—results for the pretrained model are also reported.
  • I acknowledge the distinction with models like Qwen2.5-Omni, in which the generated audio does not necessarily influence text-generation—although it can do so through the shared hidden state—but I note again that this is also not a novel aspect of your work, as Moshi already enabled such influence.
  • Given that Moshi already compared inner monologue pretraining with audio-only pretraining (Table 8 in their paper), I also don't consider a controlled comparison between these approaches as meaningfully novel.
  • I am not aware of other works that have conducted controlled comparisons between pretraining with inner monologue and text-speech interleaving pretraining, so I acknowledge this is a potential contribution. Nevertheless, given the existing empirical evidence regarding the advantages of inner monologue across many works, I am not convinced that such a comparison would constitute a sufficiently impactful contribution unless it reveals substantially new insights, which I don't think is the case here.

Regarding the value of the study of cross-modal distillation in speech-to-speech models:

  • Methodologically, I do not see a meaningful contribution, as the method remains cross-modal context distillation applied to text logits. In the absence of methodological novelty, I would expect the paper to provide novel insights that allow us to better understand the existing method. However, I do not find such insights in the presented experiments. In particular, the advantages of text-speech cross-modal distillation for cross-modal knowledge transfer, which are presented as the main results of the paper, are known from previous works. While I acknowledge that it is a nice data point to know that such advantages survive joint text-speech prediction training, I believe this was largely expected and thus I do not consider it a significantly novel contribution. Moreover, as I mentioned, works like Qwen3.5-Omni that applied cross-modal distillation to speech-to-speech generation have already hinted at these benefits in the speech-to-speech setting. I do not think that the difference in the training stage at which distillation is applied yields a meaningfully novel insight.
  • Regarding the existing experiments measuring the effects of distillation on speech generation, I consider them the most interesting part of the paper, and I appreciate the new experiment and your openness to discussion. I would like to see a more careful characterization of how cross-modal distillation affects speech generation. For instance, building on the new experiment, it would be interesting to know what is the divergence between inner-monologue content and generated content. It would also be useful to study how the different linguistic, acoustic and paralinguistic metrics scale with increased amounts of training data across different distillation weights. This would not necessarily require a large scale scaling study, and could potentially be conduced with smaller models if compute is a constraint. Given that we already observe some interactions between cross-modal context distillation and paralinguistic modeling, it is all the more interesting and important to better understand these phenomena. Likewise, another potentially valuable contribution would be to study ways to promote a better synergy between the training objectives, for example through careful loss weighting, training schedules, or data-replay.

I acknowledge that these suggestions could amount to an entirely different paper, and I think that is largely my point: I believe a more systematic investigation of the interaction between cross-modal text-speech context distillation and speech generation would be a more interesting and valuable research direction in this context. While I think the paper has some useful insights, as mentioned since my initial review, I remain unconvinced that the contributions of the current paper meet the bar for novelty and significance expected at ICLR.

Reviewer_hhTyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces TGSLM, which is initialized from Qwen3 and trained on word-aligned speech-text data. A random word boundary divides each utterance into a speech-only context and a continuation. so every future word is predicted in text before its speech segment. TGSLM-distill adds forward-KL supervision at text positions from the frozen original Qwen3 model, which conditions on the corresponding ground-truth text context. Some models additionally replay FineWeb text batches.

  1. At 0.8B parameters and nominally 21.3k hours of speech data:
  • TGSLM lowers ASR-based generative perplexity from 151.7 for the speech-only baseline to 58.9. Distillation alone lowers it to 56.9, and distillation plus text replay to 51.0.
  • On Known-1000, replay alone raises Pass@5 from 6.7% to 12.5%, while the reported best controlled distilled setting reaches 22.4%.
  1. A 1.9B model trained on 135k hours reaches 45.0 genPPL and 37.9% Known-1000 Pass@5.
  2. The paper also reports StoryCloze, UTMOS-v2, speaker similarity, and a preliminary synthetic emotion-sensitivity evaluation.
Critical strengths
  1. Large controlled linguistic-quality gain. The 151.7 to 58.9 genPPL change relative to speech-only training is substantial. Matching a Whisper-transcript-conditioned text-leading reference without supplying prompt text is an interesting result.

  2. Broad outcome coverage. The paper evaluates open-ended continuation, factual completion, StoryCloze, repetition, acoustic quality, speaker similarity, and a preliminary paralinguistic property.

  3. Readable presentation. The main mechanism is easy to follow, limitations are acknowledged, and the appendix gives useful training and decoding details.

Critical weaknesses
  1. Unmatched architecture comparison. The interleaved baseline sees roughly half as much speech per update, while TGSLM also receives transcript targets and predicts extra text tokens. The results show that the complete TGSLM recipe is stronger, but unequal exposure, supervision, and compute prevent attributing the gain specifically to text-before-speech ordering. See question 1.

  2. Missing capability-retention baselines. Because the frozen teacher and student start from the same Qwen3 checkpoint, the KL objective mainly preserves the original text behavior during speech adaptation. Without matched frozen-backbone, projector-only, adapter/LoRA, or stronger replay baselines, its advantage over standard retention strategies is unclear. Known-1000 also remains far below the text backbone, so the result is partial retention rather than factual-knowledge recovery. A stricter claim baseline is required.

  3. The source of the gains is not identified. The paper evaluates speech only after ASR transcription, without reporting the accuracy of the internal text or its agreement with the generated speech. It also evaluates open-ended generation on the intersection of prompts for which every model produces non-empty output. These choices make it difficult to separate language-model gains from speech-generation, ASR, or selection effects.

  4. Evaluation and efficiency evidence is narrow. Known-1000 and StoryCloze use one synthetic voice, and the small emotion study uses synthetic examples filtered and evaluated with the same classifier. The paper also reports no latency, throughput, memory, or real-time factor despite TGSLM's longer alternating sequence. This limits the generality for real voice and actual deployment affects (Question 4 and 5).

  5. Related-work positioning is incomplete. Established work on joint speech-text decoding and capability preservation, including the ASRU 2025 joint-decoding study, SSR, AlignFormer, DeSTA2, and VoiceTextBlender, is missing. These papers do not duplicate TGSLM, but they narrow its novelty claims.

  • H. Wu, Y. Hu, R. Fan, X. Wang, K. Kumatani, B. Ren, J. Yu, H. Lu, L. Wang, Y. Qian, and J. Li, "Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model," in Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2025, arXiv:2506.04518.
  • W. Tan, H. Inaguma, N. Dong, P. D. Tomasello, and X. Ma, "SSR: Alignment-Aware Modality Connector for Speech Language Models," in Proc. 22nd International Conference on Spoken Language Translation (IWSLT), pp. 56-75, 2025, doi:10.18653/v1/2025.iwslt-1.5.
  • Y. Peng, K. C. Puvvada, Z. Chen, P. Zelasko, H. Huang, K. Dhawan, K. Hu, S. Watanabe, J. Balam, and B. Ginsburg, "VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning," in Proc. 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5787-5802, 2025, doi:10.18653/v1/2025.naacl-long.298.
  • J. Ni, Y. Ma, W. Wang, Q. Chen, D. Ng, H. Lei, T. H. Nguyen, C. Zhang, B. Ma, and E. Cambria, "Adaptive Knowledge Distillation Between Text and Speech Pre-Trained Models," in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, doi:10.1109/ICASSP49357.2023.10096950.
  • W. I. Cho, D. Kwak, J. W. Yoon, and N. S. Kim, "Speech to Text Adaptation: Towards an Efficient Cross-Modal Distillation," in Proc. Interspeech, pp. 896-900, 2020, doi:10.21437/Interspeech.2020-1246.
  • R. Fan, Y. Wang, R. Zhao, L. Ren, K. Deng, X. Chen, A. Zare, B. Ren, Y. Hu, J. Chen, Y. Huang, Y. Shen, and J. Li, "Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving," in arXiv:2607.01733. Accepted at SLT 2026
  1. Emotion task study, many of recent studies question the fairness of the emotion tasks where the model can predict the correct answer with text-related information only. Any result/baseline that using the speech transcription input? It would be great to have the analysis that how many examples are speech-mandatory and the percentages that are not in the test data. Hence, I am concerning the claim of the limited impact on emotion sensitivity. The paper needs to careful on the conclusion as it is claimed as one of the major contributions.
Questions for authors
  1. For each architecture, how many speech frames, text targets, total predicted tokens, optimizer updates, and training FLOPs were consumed? Does "21.3k h" mean corpus size or actual exposure?

  2. On Known-1000 and StoryCloze, what accuracy does the internally generated text achieve before speech synthesis, and how often does the generated speech transcription disagree with that text?

  3. What are the per-model rates of empty output, premature termination, boundary errors, and text-speech desynchronization? How do the primary results change when failures are included instead of evaluating only the all-model success intersection?

  4. Are latency, real-time factor, throughput, and peak-memory measurements available for TGSLM and the speech-only, interleaved, and text-leading systems?

  5. Do any existing results cover natural multi-speaker speech, additional TTS systems, or standard spoken QA/reasoning benchmarks?

1 comment

We thank the reviewer for the constructive feedback. We address the main concerns and questions below.

Q1. Training comparison [Critical Weakness 1; Question 1]. The reported 21.3k hours refers to speech exposure during training. We matched the backbone and number of optimizer updates across the controlled comparisons, while varying the sequence construction. As noted in Footnote 6, the interleaved baseline sees fewer speech frames per update because its sequence budget is shared between speech and text. Thus, the comparisons are not strictly compute-matched (FLOP), but evaluate the different training formulations under similar settings.

Q2. Internal text accuracy and consistency [Critical Weakness 3; Question 2]. We measured WER between internally generated text and ASR-transcribed speech on the Emilia validation set (storyCloze and known-1000 follow a similar trend).

Distillation weight λ00.10.250.50.75
WER (%)8.19.512.423.339.0

Stronger distillation improves factual knowledge retention (Fig. 2) but increases disagreement between internal text and generated speech, revealing a trade-off between knowledge retention and text-speech consistency.

Q3. Generation failures [Critical Weakness 3; Question 3]. As reported in Table 8, we evaluate on a shared subset of 1,325 out of 1,500 prompts, covering 4,957 out of 7,500 continuations. While this excludes unsuccessful generations, we believe evaluating all models on the same subset provides a fairer comparison than using different subsets for each model, which could introduce additional confounding factors due to differences in sample difficulty. Our TGSLM-distill has a 0.72% empty rate, while Mimo has 5% empty rate and Llama-mimi has 2.12% so we did not take advantage of the shared subset, but use this to calibrate the genPPL numbers.

Q4. Efficiency [Critical Weakness 4; Question 4]. We did not evaluate inference latency or throughput, as our focus is the modeling formulation rather than inference optimization.

Q5. Generalization [Critical Weaknesses 4–5; Question 5]. Our open-ended continuation evaluation uses natural speech from Emilia. Many existing speech-text approaches focus on specific downstream tasks, and it remains unclear whether their improvements generalize to open-ended speech generation.

Additional clarification on retention baselines and related work [Critical Weaknesses 2 and 5]. Frozen-backbone and adapter-based approaches address complementary questions and could potentially be combined with our method, but they are not directly comparable. Our goal is to study whether cross-modal distillation improves factual knowledge retention during speech-to-speech pretraining, as demonstrated by our controlled distillation ablations (Fig. 2). We appreciate the suggested references and will expand the related-work discussion to better position our contribution, but we emphasize that they address a different set of questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.