- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper compares text-first continued pretraining (CPT) with joint text-audio pretraining from scratch (SPT), accounting for text-backbone pretraining within a fixed total token budget. With matched cumulative text and audio token allocations, SPT generally achieves a better text-audio performance trade-off across the evaluated settings. Additional experiments examine transition-specific optimization, model scale, MoE architectures, speech generation, and downstream speech question answering.
Critical strengths
-
The budget accounting is well motivated. Matching both total training tokens and cumulative modality allocations provides a useful comparison of training curricula and avoids treating text-backbone pretraining as a free resource.
-
The paper examines different budgets, mixture ratios, model sizes, and optimization settings, with additional multi-seed experiments. The learning-rate ablation is useful because it reveals substantial sensitivity in the CPT baseline.
Critical weaknesses
-
Figure 2 shows that transition-specific optimization substantially narrows the CPT-SPT gap. However, the multi-seed results in Table 3 only evaluate the default CPT setup. It therefore remains unclear whether the smaller gap between SPT and tuned CPT is consistent across random seeds. Reporting multi-seed results for the competitive CPT variants would strengthen the central conclusion.
-
The comparison does not explore when audio should be introduced under a fixed budget. The main matched-budget experiments devote half of CPT's total training tokens to text-only pretraining. It remains unclear whether a shorter text-only stage, followed by a longer mixed stage with the same cumulative text/audio token counts, could close the gap with SPT.
-
The study primarily establishes a recipe preference for small-scale, discrete-token English speech models. Its extensions add breadth but do not resolve the baseline and curriculum-selection concerns. Moreover, ASR and transcription-based TTS evaluation provide limited evidence about broader audio understanding or speech generation quality.
Questions for authors
NA
5 comments
2. Sensitivity to the Text-Only Pretraining Duration (W2)
To test whether a shorter text-only stage could close the gap with SPT, we reduced text-only pretraining from 30B to 15B tokens and extended mixed CPT from 30B to 45B tokens, keeping the total 60B-token budget and cumulative text–audio allocations fixed. This improves CPT’s mean text and audio performance and reduces variation in audio performance across seeds at every allocation. SPT still achieves higher mean text performance at all five allocations and higher mean audio performance at three, with trade-offs at the remaining two. Thus, earlier audio introduction strengthens CPT, but SPT remains generally competitive.
The original 30B-token text-only budget followed the Chinchilla-style token-to-parameter ratio, with an equal budget assigned to mixed CPT. The resulting 50% split was not selected as the optimal switch point.
Both CPT settings use the Default CPT transition strategy. Below, 30B and 15B denote the text-only pretraining budgets. All results are mean ± standard deviation across three seeds.
| Text:Audio | 30B CPT Text | 30B CPT Audio | 15B CPT Text | 15B CPT Audio | SPT Text | SPT Audio |
|---|---|---|---|---|---|---|
| 39:21 | 43.07 ± 0.38 | 48.60 ± 10.83 | 44.73 ± 0.57 | 85.60 ± 0.36 | 46.05 ± 0.49 | 85.50 ± 0.42 |
| 42:18 | 44.00 ± 0.17 | 76.77 ± 8.07 | 44.50 ± 0.28 | 84.05 ± 1.20 | 45.23 ± 0.25 | 84.63 ± 1.00 |
| 45:15 | 44.70 ± 0.50 | 77.83 ± 6.48 | 45.07 ± 0.67 | 83.10 ± 0.61 | 45.77 ± 0.61 | 84.43 ± 0.32 |
| 48:12 | 45.13 ± 0.21 | 62.97 ± 29.92 | 45.45 ± 0.07 | 82.70 ± 0.71 | 46.10 ± 0.20 | 81.97 ± 0.42 |
| 51:9 | 45.57 ± 0.06 | 43.53 ± 19.40 | 45.75 ± 0.21 | 78.60 ± 1.56 | 46.57 ± 0.35 | 80.80 ± 0.61 |
3. Scope of Generalization and Audio Evaluation (W3)
We agree that ASR and transcription-based TTS evaluation alone cannot establish broad audio understanding or overall speech generation quality. Similar to Moshi [1] and Qwen2.5-Omni [2], our TTS extension uses WER computed from transcriptions of generated speech to evaluate how accurately it conveys the target text. The purpose is to test whether differences between training strategies also appear in content accuracy, rather than to evaluate naturalness, audio quality, or speaker similarity. We will clarify this purpose and evaluation scope in the paper.
Reference
- [1] Moshi: a speech-text foundation model for real-time dialogue, 2024
- [2] Qwen2.5-Omni Technical Report, 2025
We thank the reviewer for the constructive feedback, which has helped us clarify our claims and strengthen the empirical evidence.
We thank the reviewer for the careful review and constructive feedback, which helped us strengthen the robustness of our experimental evaluation.
1. Multi-Seed Results Across Experimental Settings (W1)
We compared SPT with transition-tuned CPT across three seeds. In the 60B-token setting, SPT achieves higher mean text and audio performance than Default CPT, LR-Matched CPT, and Low-LR CPT at all five allocations. Transition tuning improves CPT performance and stability, but SPT retains its advantage in mean performance. The overall trend against Default CPT also holds across the other training budgets and model scales.
The original Table 3 evaluated only Default CPT. We added two CPT transition strategies in the 60B-token setting and extended the 200B-token and smaller-model comparisons in Figure 1 to three seeds each. All values are mean ± standard deviation. Bold values indicate the highest mean performance for each modality at each allocation.
60B-token setting
200B-token setting
Smaller-model setting