- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper systematically studies joint speech-text pre-training from scratch. Specifically, the authors examine how model scale, capacity allocation, and modality-specific supervision affect ASR and text modeling performance. Under fixed compute budgets, increasing acoustic capacity or increasing the ASR supervision share does not necessarily improve recognition, while text supervision can benefit ASR. The experiments also show that bidirectional attention over the audio prefix improves encoder-free models. Finally, the paper develops and evaluates a five-factor scaling law that predicts speech/text losses from model capacity and modality-specific exposure.
Critical strengths
- The paper presents extensive controlled experiments across different model sizes, training budgets, and speech-text mixtures. These experiments provide useful evidence about the resource trade-offs and help guide the development of joint speech-text models.
- The five-factor scaling law predicts speech/text losses more accurately than the baseline measures. Its performance on model configurations and training budgets excluded from fitting makes its predictive ability convincing.
Critical weaknesses
- It is unclear how well the findings apply to other training setups.
- The paper reports how much data the models use, but gives limited information about the corpus, including the amount of unique speech, language composition, how transcripts were produced, etc. These details are important for reproducing the experiments and for the readers to use the scaling law in their model development.
- The same learning rate is used across model sizes, while batch sizes change across data mixtures. A sensitivity analysis would help support whether the main findings hold under other training settings.
- The paper is difficult to follow. Frequent switches between models, compute budgets, and experimental setups make it hard to track the comparisons. Some important details, including the evaluation protocol, are left in the appendices, which makes it difficult to connect the evidence to the main claims.
- A potential benefit of native speech-text pretraining is learning information beyond linguistic content. However, ASR supervision primarily focuses on transcript prediction and does not explicitly target paralinguistic cues or speaker characteristics. The experiments therefore leave unclear whether native training benefits tasks that depend on this information and whether the reported scaling trends extend to such tasks.
Questions for authors
- What are the unique speech-corpus size, language composition, and transcription-generation procedures? Please provide details for the corpus used for the experiments.
- Beyond the widths and layer counts, please provide the complete architecture settings, e.g., head counts, FFN dimensions, and positional encoding.
5 comments
5. Architecture settings.
The frontend has 128 mel bins at 100 Hz and three stride-two causal convolutional layers, producing 12.5 Hz acoustic features. The standard encoder is bidirectional; the decoder is causal.
| Model | Parameters (M) | Width | Encoder / decoder layers | Q / KV heads | Encoder / decoder FFN |
|---|---|---|---|---|---|
| T | 149.16 | 512 | 9 / 12 | 8 / 2 | 2048 / 1536 |
| S | 327.21 | 768 | 12 / 16 | 12 / 2 | 3072 / 2304 |
| B | 462.63 | 896 | 14 / 18 | 14 / 2 | 3584 / 2688 |
| M | 794.41 | 1152 | 16 / 21 | 18 / 2 | 4608 / 3456 |
| L | 1449.05 | 1536 | 18 / 23 | 24 / 2 | 6144 / 4608 |
| XL | 2723.49 | 2048 | 20 / 26 | 32 / 2 | 8192 / 6144 |
Q/KV counts apply to both encoder and decoder. Both use 64-dimensional heads, SwiGLU and RoPE with base 10⁶. The encoder uses pre-RMSNorm (epsilon 10⁻⁶) and Q/K normalization; the decoder is Qwen3-style. The adapter is LayerNorm plus a bias-free linear projection and an additive audio-modality embedding. Input/output text weights are tied, with 151,936 embedding rows. The decoder-position cap is 8,192 per sample, distinct from aggregate pack budgets.
6. How to read the comparisons.
The balanced sweep varies size at fixed compute; the L sweep redistributes approximately fixed non-embedding capacity; the M grid crosses capacity allocation and speech/text mixture. The fitted responses are validation NLLs. External recognition uses greedy micro CER/WER, with the 30-second LibriSpeech selection; Core-v2 averages chance-adjusted accuracies on 22 text tasks. Those recognition scores test pretrained checkpoints directly, unlike the new task-specific adaptations above. Held-out prediction concerns configurations and budgets within the studied training distribution, not unchanged coefficients under arbitrary data or optimization changes.
Thank you for the additional experiments and clarifications. The learning-rate and batch-size checks strengthen the comparisons they cover, and the emotion probe provides useful evidence beyond transcription. The architecture table answers my configuration question. I believe these results and the comparison guide are important to include in the paper.
Readability remains my main concern. Please also clarify how the transcripts were produced.
Thank you for the specific questions. We report the new optimization controls, speech-transfer evidence, corpus description and complete architecture settings here.
1. Learning-rate sensitivity.
The main grid compares a common AdamW recipe with full-horizon cosine decay and peak LR 6 × 10⁻⁴. We tested half and twice this rate at S (327M) and XL (2.72B), both at 1 C₀, retaining geometry, seed, data, packing and training horizon. Last scheduled validation losses are:
At both scales, the original LR gives the lowest ASR NLL of the three, while the higher LR improves text NLL. Thus a single LR change does not improve both objectives. These checks characterize sensitivity at the sampled scales, rather than establish globally optimal hyperparameters.
2. Batch sensitivity at matched exposure.
We doubled gradient accumulation for two B (463M) mixtures and approximately halved optimizer updates, retaining the same data streams, seed, peak LR and complete cosine schedule. Text and ASR exposure match at paired validation points; full-horizon rounding is below 0.004%. The loss remains the sum of separately normalized ASR and text means with unit weights.
The larger batch delays the initial ASR-loss decline but lowers final validation NLLs. External scores are mixed. Importantly, both batch settings retain the same trade-off: higher ASR share favors recognition, while lower share favors text NLL and Core-v2. These are two single-seed sensitivity controls, not evidence of universal batch invariance. The higher-share original ran on H200 and its repeat on H100; that pair is not hardware-identical.
3. Information beyond transcription.
We tested frozen acoustic features from S/4C₀ on four-class IEMOCAP emotion recognition, using the same architecture at random initialization as a control. The encoder receives only audio. A linear readout is selected on a separate development session, then refitted on the four non-test sessions; five held-out-session folds separate speakers.
UAR is mean recall across the four classes; ± is sample SD across sessions. All 5,531 utterances and both feature taps are retained, with final-layer improvement in every fold. This demonstrates useful emotion-related information under supervised readout; it does not isolate prosody from lexical cues.
We also completed English→Chinese speech translation and spoken-question QA with S/4C₀ versus random initialization. Both receive three epochs of identical full-model adaptation, LR 10⁻⁴, batch 64, and development-NLL selection. On a CoVoST 2 subset matched to available Common Voice 15 audio, BLEU improves from 0.87 to 29.77 (15,522 test clips; Chinese-tokenized SacreBLEU). On HeySQuAD-human, the input is a spoken question plus written passage, without the question transcript. Prespecified answerable-question F1 improves from 2.10% to 42.24%; full-set F1 is 52.09%→54.53%, while no-answer accuracy is 67.97%→58.43%. These establish supervised transfer at one checkpoint, not scaling trends for emotion, translation or QA.
4. Corpus details and reproducibility.
The speech pool contains approximately 1.5 million hours of Internet speech–transcript data, predominantly Chinese and English. Clips are capped at 30 seconds and average about 17 seconds. A transcript-based random-sample audit suggests roughly comparable audio duration in the two languages. The text pool contains roughly one trillion supervised targets, including EOS, from DCLM, Nemotron and Dolma-derived sources. Corpus inventory and cumulative run exposure are distinct.
The speech pipeline includes cleaning, annotation and selection; source-level inventories and detailed annotation recipes remain private. Public ASR benchmark training splits are not added to pretraining. The numerical materials support reproduction of the analyses, while original training replication requires access to the private corpus.