acceptodds
Under review as a conference paper at ICLR 2027

The Moving Frontier of Native Speech–Language Pretraining

Abstract

Multimodal large models are typically built through a late-fusion recipe: language models and modality-specific components are pretrained separately, then connected and jointly trained. This approach builds on advances in unimodal pretraining and has become a standard route to multimodal capabilities. Recently, frontier systems have begun to explore a different recipe, learning perception and language jointly from scratch. These developments suggest that native multimodal pretraining is a viable alternative and motivate a closer examination of the principles governing its scaling. In this work, we study this paradigm through continuous speech and text, using speech–language modeling as a concrete setting in which to investigate how model capacity, training compute, and modality-specific supervision interact. Across over 140 completed configurations spanning 149M–2.72B parameters, we demonstrate that randomly initialized models can jointly acquire speech recognition and text-modeling capabilities. We characterize how scale, capacity placement, and supervision shape speech and text learning, and use boundary and initialization controls to probe these relationships. A scaling law connects these empirical findings to loss prediction across configurations. Together, these results provide an empirical foundation for native speech–language pretraining and a quantitative guide to allocating its training resources.

Primary area:
foundation or frontier models, including LLMs
Keywords:
native multimodal pretraining, scaling laws, speech-language modeling, compute-efficient training, capacity allocation, data mixture

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_it1SOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper systematically studies joint speech-text pre-training from scratch. Specifically, the authors examine how model scale, capacity allocation, and modality-specific supervision affect ASR and text modeling performance. Under fixed compute budgets, increasing acoustic capacity or increasing the ASR supervision share does not necessarily improve recognition, while text supervision can benefit ASR. The experiments also show that bidirectional attention over the audio prefix improves encoder-free models. Finally, the paper develops and evaluates a five-factor scaling law that predicts speech/text losses from model capacity and modality-specific exposure.

Critical strengths
  • The paper presents extensive controlled experiments across different model sizes, training budgets, and speech-text mixtures. These experiments provide useful evidence about the resource trade-offs and help guide the development of joint speech-text models.
  • The five-factor scaling law predicts speech/text losses more accurately than the baseline measures. Its performance on model configurations and training budgets excluded from fitting makes its predictive ability convincing.
Critical weaknesses
  • It is unclear how well the findings apply to other training setups.
    • The paper reports how much data the models use, but gives limited information about the corpus, including the amount of unique speech, language composition, how transcripts were produced, etc. These details are important for reproducing the experiments and for the readers to use the scaling law in their model development.
    • The same learning rate is used across model sizes, while batch sizes change across data mixtures. A sensitivity analysis would help support whether the main findings hold under other training settings.
  • The paper is difficult to follow. Frequent switches between models, compute budgets, and experimental setups make it hard to track the comparisons. Some important details, including the evaluation protocol, are left in the appendices, which makes it difficult to connect the evidence to the main claims.
  • A potential benefit of native speech-text pretraining is learning information beyond linguistic content. However, ASR supervision primarily focuses on transcript prediction and does not explicitly target paralinguistic cues or speaker characteristics. The experiments therefore leave unclear whether native training benefits tasks that depend on this information and whether the reported scaling trends extend to such tasks.
Questions for authors
  • What are the unique speech-corpus size, language composition, and transcription-generation procedures? Please provide details for the corpus used for the experiments.
  • Beyond the widths and layer counts, please provide the complete architecture settings, e.g., head counts, FFN dimensions, and positional encoding.
5 comments

Thank you for the specific questions. We report the new optimization controls, speech-transfer evidence, corpus description and complete architecture settings here.

1. Learning-rate sensitivity.

The main grid compares a common AdamW recipe with full-horizon cosine decay and peak LR 6 × 10⁻⁴. We tested half and twice this rate at S (327M) and XL (2.72B), both at 1 C₀, retaining geometry, seed, data, packing and training horizon. Last scheduled validation losses are:

ModelPeak LRASR NLLText NLL
S3 × 10⁻⁴0.21172.4110
S6 × 10⁻⁴0.19842.3875
S1.2 × 10⁻³0.20002.3761
XL3 × 10⁻⁴0.34952.4973
XL6 × 10⁻⁴0.34122.4755
XL1.2 × 10⁻³0.37082.4506

At both scales, the original LR gives the lowest ASR NLL of the three, while the higher LR improves text NLL. Thus a single LR change does not improve both objectives. These checks characterize sensitivity at the sampled scales, rather than establish globally optimal hyperparameters.

2. Batch sensitivity at matched exposure.

We doubled gradient accumulation for two B (463M) mixtures and approximately halved optimizer updates, retaining the same data streams, seed, peak LR and complete cosine schedule. Text and ASR exposure match at paired validation points; full-horizon rounding is below 0.004%. The loss remains the sum of separately normalized ASR and text means with unit weights.

ASR target shareBatchASR NLLText NLLAISHELL CER (%)Libri clean / other WER (%)Core-v2
18.2%Original0.26002.52288.0625.471 / 13.5410.0912
18.2%Doubled0.24912.48767.6995.589 / 13.5140.0880
46.1%Original0.23392.71796.8194.851 / 12.5800.0597
46.1%Doubled0.22932.63366.8625.187 / 12.7720.0752

The larger batch delays the initial ASR-loss decline but lowers final validation NLLs. External scores are mixed. Importantly, both batch settings retain the same trade-off: higher ASR share favors recognition, while lower share favors text NLL and Core-v2. These are two single-seed sensitivity controls, not evidence of universal batch invariance. The higher-share original ran on H200 and its repeat on H100; that pair is not hardware-identical.

3. Information beyond transcription.

We tested frozen acoustic features from S/4C₀ on four-class IEMOCAP emotion recognition, using the same architecture at random initialization as a control. The encoder receives only audio. A linear readout is selected on a separate development session, then refitted on the four non-test sessions; five held-out-session folds separate speakers.

Prespecified featureRandom UAR (%)Pretrained UAR (%)
Final encoder layer57.73 ± 1.4071.55 ± 2.46
Encoder layer 657.16 ± 2.6371.01 ± 3.32

UAR is mean recall across the four classes; ± is sample SD across sessions. All 5,531 utterances and both feature taps are retained, with final-layer improvement in every fold. This demonstrates useful emotion-related information under supervised readout; it does not isolate prosody from lexical cues.

We also completed English→Chinese speech translation and spoken-question QA with S/4C₀ versus random initialization. Both receive three epochs of identical full-model adaptation, LR 10⁻⁴, batch 64, and development-NLL selection. On a CoVoST 2 subset matched to available Common Voice 15 audio, BLEU improves from 0.87 to 29.77 (15,522 test clips; Chinese-tokenized SacreBLEU). On HeySQuAD-human, the input is a spoken question plus written passage, without the question transcript. Prespecified answerable-question F1 improves from 2.10% to 42.24%; full-set F1 is 52.09%→54.53%, while no-answer accuracy is 67.97%→58.43%. These establish supervised transfer at one checkpoint, not scaling trends for emotion, translation or QA.

4. Corpus details and reproducibility.

The speech pool contains approximately 1.5 million hours of Internet speech–transcript data, predominantly Chinese and English. Clips are capped at 30 seconds and average about 17 seconds. A transcript-based random-sample audit suggests roughly comparable audio duration in the two languages. The text pool contains roughly one trillion supervised targets, including EOS, from DCLM, Nemotron and Dolma-derived sources. Corpus inventory and cumulative run exposure are distinct.

The speech pipeline includes cleaning, annotation and selection; source-level inventories and detailed annotation recipes remain private. Public ASR benchmark training splits are not added to pretraining. The numerical materials support reproduction of the analyses, while original training replication requires access to the private corpus.

5. Architecture settings.

The frontend has 128 mel bins at 100 Hz and three stride-two causal convolutional layers, producing 12.5 Hz acoustic features. The standard encoder is bidirectional; the decoder is causal.

ModelParameters (M)WidthEncoder / decoder layersQ / KV headsEncoder / decoder FFN
T149.165129 / 128 / 22048 / 1536
S327.2176812 / 1612 / 23072 / 2304
B462.6389614 / 1814 / 23584 / 2688
M794.41115216 / 2118 / 24608 / 3456
L1449.05153618 / 2324 / 26144 / 4608
XL2723.49204820 / 2632 / 28192 / 6144

Q/KV counts apply to both encoder and decoder. Both use 64-dimensional heads, SwiGLU and RoPE with base 10⁶. The encoder uses pre-RMSNorm (epsilon 10⁻⁶) and Q/K normalization; the decoder is Qwen3-style. The adapter is LayerNorm plus a bias-free linear projection and an additive audio-modality embedding. Input/output text weights are tied, with 151,936 embedding rows. The decoder-position cap is 8,192 per sample, distinct from aggregate pack budgets.

6. How to read the comparisons.

The balanced sweep varies size at fixed compute; the L sweep redistributes approximately fixed non-embedding capacity; the M grid crosses capacity allocation and speech/text mixture. The fitted responses are validation NLLs. External recognition uses greedy micro CER/WER, with the 30-second LibriSpeech selection; Core-v2 averages chance-adjusted accuracies on 22 text tasks. Those recognition scores test pretrained checkpoints directly, unlike the new task-specific adaptations above. Held-out prediction concerns configurations and budgets within the studied training distribution, not unchanged coefficients under arbitrary data or optimization changes.

Thank you for the additional experiments and clarifications. The learning-rate and batch-size checks strengthen the comparisons they cover, and the emotion probe provides useful evidence beyond transcription. The architecture table answers my configuration question. I believe these results and the comparison guide are important to include in the paper.

Readability remains my main concern. Please also clarify how the transcripts were produced.

Reviewer_gsxjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper explores the effectiveness of natively pretraining a SpeechLLM directly from scratch by learning speech processing and language modeling abilities without the need for specific speech encoder and LLM pretraining and, later, modality fusion. 140 experiments are performed on an internal English-only corpus over different model sizes, and a scaling low is presented to connect the empirical findings.

Critical strengths
  • The native speech-text pretraining analyzed in the paper is very interesting and relatively understudied, at least at this scale.
  • The paper is very well written, and easy to follow in its motivation but becomes more difficult in the results part.
  • The various analyses are interesting, resulting in a potential continuous-feature SpeechLLM scaling law.
Critical weaknesses
  • The focus of the paper is to systematically study the native speech-text pretraining following the practice of frontier models. However, frontier models are not trained on English corpora only, making this study still pretty distant from what constitutes a real data mixture for a strong/frontier SpeechLLM, or more broadly, MLLM. There is no guarantee that the identified scaling law also works for the real multilingual scenario.
  • Several tasks are used for LLM evaluation, but the speech modeling part is tested on ASR only. This is, again, pretty distant from a frontier MLLM, trained and used on generic instruction following abilities.
  • The study is done on an internal corpus, making its reproducibility impossible.
  • Some results are not surprising, for instance, that a deep acoustic encoder is important (e.g., Whisper, wav2vec 2.0, and SpeechT5 papers), or that text supervision can also help speech (e.g., "LLM can Read Spectrogram: Encoder-free Speech-Language Modeling" by Fan et al., or "Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens" by Manakul et al.). But these results are not positioned in the literature nor discussed compared to existing works.
Questions for authors
  • What is the total size of the training set? What is the domain of the internal training pool? The data details are severly underspecified
  • What happens when scaling to a multilingual (real) case to the scaling laws?
  • What is the performance of the model on tasks outside ASR? How does the training generalize on very common tasks such as spoken question answering, summarization, and translation?
4 comments

Thank you for the questions about data, task coverage and positioning. We provide the relevant details and newly completed experiments directly below.

1. The speech corpus is predominantly Chinese and English, not English-only.

It contains approximately 1.5 million hours of Internet speech–transcript data. Clips are at most 30 seconds long and average about 17 seconds. Transcript-based language identification on 20,000 randomly sampled segments suggests roughly comparable Chinese and English audio duration. The auxiliary text pool provides roughly one trillion supervised targets, including document EOS, from DCLM, Nemotron and Dolma-derived sources.

The speech is collected from the Internet and processed through cleaning, annotation and selection, rather than assembled from public ASR benchmark training splits. Source-level inventories and detailed annotation recipes remain private. These figures describe the corpus inventory, not cumulative exposure in a particular run. The numerical supplement supports analysis reproduction; reproducing the original training also requires corpus access.

2. What generalization does the law test?

Our held-out tests vary model configurations and compute budgets within this bilingual training setting. They do not change the pretraining language proportions, so transfer of unchanged coefficients to another language mixture remains untested. A useful distinction is between generalizing across configurations and generalizing across data distributions. The present evidence addresses the former; it is not an English-only experiment or a claim to reproduce a frontier model's full mixture.

3. New speech tasks beyond transcription.

We completed two downstream tasks using the paper's S/4C₀ checkpoint (327M parameters) and a randomly initialized model of the same architecture:

Task and metricRandom initializationS/4C₀ pretrained
Speech translation, BLEU / chrF0.87 / 2.6029.77 / 24.34
Spoken QA, answerable EM / F1 (%)1.70 / 2.1035.53 / 42.24

Both initializations receive the same task data and three epochs of full-model adaptation: fresh AdamW, peak LR 10⁻⁴, global batch 64, and development-NLL checkpoint selection. Each task starts independently from its original initialization. These single-seed comparisons measure supervised transfer under a common downstream recipe, not zero-shot instruction following.

Translation is English speech to Chinese text, using original CoVoST 2 split IDs matched to available Common Voice 15 audio: 289,257 training and 15,522 test clips. This is a retained local-audio subset, not the full original CV4 benchmark. BLEU uses SacreBLEU 2.5.1 with Chinese tokenization.

HeySQuAD-human supplies a spoken question and a written passage; question transcripts are not provided to the model. We reserve development passages from the training split and use the author's validation split for testing. The primary EM/F1 above was specified in advance on its 1,002 answerable questions. Across all 4,158 questions, F1 is 52.09→54.53; no-answer accuracy is 67.97→58.43, so the gain is specifically in answering questions rather than abstention. On the same 512 answerable diagnostic examples, pretrained-model F1 falls from 42.09 with correct audio to 1.96 with mismatched audio and 1.43 with silence, supporting actual use of the spoken question.

We additionally tested audio-only frozen features on four-class IEMOCAP emotion recognition. A development-selected linear readout gives 71.55 ± 2.46% UAR (mean recall across classes) from the final pretrained encoder layer versus 57.73 ± 1.40% from the random encoder. The five session-held-out folds separate speakers; ± denotes session-fold SD. A prespecified layer-6 check gives 71.01% versus 57.16%, consistent with the primary result.

These experiments supply direct transfer evidence beyond ASR. Summarization has not been evaluated, and these single-checkpoint studies do not establish a scaling law for the new tasks.

4. What the joint study adds to earlier findings.

We agree that encoder usefulness and benefits from text supervision have precedents, including SpeechT5 and the encoder-free and SODA studies you identify. Our contribution is the controlled, quantitative account of how these factors interact under finite compute, with separate speech and language responses and prediction on unseen configurations.

In particular, adding text at fixed ASR exposure is different from reallocating a fixed compute budget toward text. Our boundary controls test the former; the mixture grid tests the latter. Likewise, the allocation experiments ask how much acoustic capacity is useful at each budget while also preserving text capability, rather than whether an encoder can help at all. This complements the encoder-heavy preference in Li et al. and the multilingual recognition/translation scaling studied by OWLS. These distinctions explain why the individual qualitative observations and the joint scaling analysis are different contributions.

I thank the authors for their reply.

Regarding the language coverage, I suggest the authors to be more clear in the text as the fact that Chinese is covered is not explicitly stated when presenting the datasets (in fact, "Chinese" can be found only once in the text, in the "Protocol." paragraph). Even though two languages are covered, I still believe that this is far from the real multilingual settings of frontier model trainings, therefore the impact of these findings are extremely reduced. Moreover, being a closed dataset, the concern about reproducibility still applies (but I suggest to add at least the numbers presented in this rebuttal to the paper, so that a reader knows how much data we are talking about).

Regarding the results on different tasks, I struggle to see what the authors want to demonstrated between comparing a randomly initialized model and S/4C₀ pretrained. My concern is more general, as results are reported throughout the paper on ASR only, while again this is only one, although the most common, task that can be performed with a speech model. As authors mentioned in their rebuttal, the new experiments do not establish a scaling law for the new tasks, while this is exactly was I was trying to understand with my question.

For these reasons, I still believe my original assessment for the paper is correct, even if there are large margin of improvement.

Thank you for the follow-up. Your comments help us clarify both the scope of the study and the purpose of the additional experiments.

1. How we view the scope and significance of the findings

We appreciate your concern about relevance to richly multilingual frontier systems. In our title, "frontier" means the compute-conditioned speech–text Pareto frontier within the studied design space. Frontier-model developments motivate the native-training paradigm, while our contribution is a controlled study of its capacity–exposure trade-offs and prediction on held-out configurations.

Broader language coverage would strengthen this evidence. At the same time, we hope that limited coverage need not by itself imply very limited scientific value. For context, Chinchilla explicitly reports English-data training while studying compute-optimal allocation. (Hoffmann et al., 2022) In speech, wav2vec 2.0 developed and evaluated its representation-learning framework on LibriSpeech/Libri-Light. (Baevski et al., 2020) We hope our focused study can likewise offer useful foundations for broader investigations.

2. What the additional task experiments were intended to demonstrate

We apologize that our previous reply did not clearly explain the purpose of the pretrained-versus-random comparison. It tests whether ASR-plus-text pretraining learns representations that transfer to other speech tasks under a common downstream adaptation protocol. When we wrote that the experiments "do not establish a scaling law for the new tasks," we were distinguishing this transfer question from the further question of how downstream performance changes with pretraining resources.

The main pretraining sweep uses ASR and text modeling partly because of data scalability: we had access to speech–transcript pairs at approximately the million-hour scale, but not comparably large, quality-matched task-specific supervision for translation or spoken QA. The smaller downstream datasets allow us to evaluate transfer after adaptation.

To investigate the resource-dependent behavior you ask about, we have now evaluated 20 pretrained configurations, forming a complete grid of five model sizes and four pretraining budgets, on English-to-Chinese speech translation. We use the same CoVoST 2 subset described previously, with original split IDs matched to available Common Voice 15 audio.

All configurations use the same downstream data and three-epoch adaptation recipe, with checkpoint selection by development NLL. The columns below denote pretraining compute; downstream training follows a common recipe rather than equal FLOPs across model sizes.

Speech translation BLEU, higher is better

Model size0.5C₀1C₀2C₀4C₀
T — 149M19.4121.3422.6924.58
S — 327M20.4824.8928.6229.77
B — 463M19.0125.1927.7229.60
M — 794M16.2921.0326.3329.74
L — 1.45B11.7513.6322.1829.01

Increasing pretraining compute improves translation BLEU within every evaluated size. Larger models do not uniformly perform better at a fixed pretraining budget. For example, S exceeds L by about 11 BLEU at 1C₀, but by less than 1 BLEU at 4C₀.

We also evaluated three capacity allocations at approximately 0.79B parameters, 1.5C₀, and the same pretraining data mixture. BLEU is 20.64 / 24.63 / 27.02 for decoder-heavy / balanced / encoder-heavy configurations. This extends the transfer comparison to the effects of capacity allocation.

Together with the previously reported emotion probe, these results support transfer beyond transcription in the evaluated tasks. The translation grid additionally shows resource-dependent transfer trends.

We will make the distinction between pretraining scaling, downstream transfer, and language coverage clearer in the paper. Thank you again for your helpful comments and continued engagement. We would be happy to provide further explanations or additional experimental results if they would help address any remaining questions.

Reviewer_8HbFOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents an analyses of the scaling properties of natively multi-modal (speech + text) pre-training. The authors train the speech encoder + Transformer backbone from scratch on web-scale ASR + text LM data across various configurations, and show how different model configurations affect scalability.

Critical strengths
  1. The paper conducts an in-depth analysis of many architecture configurations and their effects on performance, comparing different budget allocations (eg. more decoder layers and less encoder layers), the effects of data mixture ratios between speech and text, and attention masking.
Critical weaknesses
  1. The paper is very difficult to read, with many overly compressed sentences and abbreviations. It takes multiple passes to understand the experimental setup and results. Much of the necessary context is hidden in the appendix.
  2. The validation NLL increases for all setups in Figure 2 as model size increases, suggesting over-fitting and/or improper experimental setup for scaling law experiments. This (and the fact that the training data is private internal data) makes the proposed scaling curves and analyses significantly less useful for the research community.
  3. The paper's headline is native speech training. However, the models are only ever evaluated on ASR, and not other speech tasks, which is one of the main benefits of native multimodal pre-training instead of just doing ASR training. In this sense, the paper is arguably an ASR+text training version of [1,2,3] (which in this case, should be cited and mentioned in related work since the paper is effectively about ASR scaling).

[1] Li, Bo, et al. "Scaling end-to-end models for large-scale multilingual ASR.". ASRU 2021.

[2] Xiao, Alex, et al. "Scaling asr improves zero and few shot learning.". Interspeech 2022.

[3] Chen, William, et al. "OWLS: Scaling laws for multilingual speech recognition and translation models.". ICML 2025.

Questions for authors
  1. Why do the models get worse as parameter size increases?
4 comments

Thank you for the detailed comments. We clarify Figure 2's comparison first, then report new evidence beyond transcription.

1. Figure 2 holds compute fixed, not data.

A larger model costs more per update. Within the same compute budget, it therefore receives fewer updates and fewer training targets. Its additional capacity can be outweighed by the reduction in exposure; increasing model size need not monotonically reduce loss.

This is visible in our measured endpoints, independently of the fitted curves:

Compute budgetModelASR target exposure (billions)Validation ASR NLL
0.5 C₀T, 149M2.3680.2644
0.5 C₀L, 1.45B0.2860.3774
4 C₀T, 149M18.9630.2081
4 C₀L, 1.45B2.2910.1549

At the lower budget, L receives roughly one eighth of T's ASR exposure and performs worse. At the higher budget, the ordering reverses. Both models improve with more compute. Each endpoint has its own complete learning-rate schedule, rather than being cut from a longer run.

Such fixed-compute loss minima are established in the IsoFLOP literature: Hoffmann et al., §3.2/Figure 3 and Llama 3, §3.2.1/Figure 2, the latter explicitly using held-out validation NLL. An increasing branch alone is therefore not evidence of overfitting or an invalid scaling setup.

2. New experiments beyond ASR.

We completed two downstream tasks using the paper's S/4C₀ checkpoint (327M parameters) and a randomly initialized model of the same architecture:

Task and metricRandom initializationS/4C₀ pretrained
Speech translation, BLEU / chrF0.87 / 2.6029.77 / 24.34
Spoken QA, answerable EM / F1 (%)1.70 / 2.1035.53 / 42.24

Both initializations receive the same task data and three epochs of full-model adaptation: fresh AdamW, peak LR 10⁻⁴, global batch 64, and development-NLL checkpoint selection. Each task starts independently from its original initialization. These single-seed comparisons measure supervised transfer under a common downstream recipe, not zero-shot instruction following.

Translation is English speech to Chinese text, using original CoVoST 2 split IDs matched to available Common Voice 15 audio: 289,257 training and 15,522 test clips. This is a retained local-audio subset, not the full original CV4 benchmark. BLEU uses SacreBLEU 2.5.1 with Chinese tokenization.

HeySQuAD-human supplies a spoken question and a written passage; question transcripts are not provided to the model. We reserve development passages from the training split and use the author's validation split for testing. The primary EM/F1 above was specified in advance on its 1,002 answerable questions. Across all 4,158 questions, F1 is 52.09→54.53; no-answer accuracy is 67.97→58.43, so the gain is specifically in answering questions rather than abstention. On the same 512 answerable diagnostic examples, pretrained-model F1 falls from 42.09 with correct audio to 1.96 with mismatched audio and 1.43 with silence, supporting actual use of the spoken question.

We also evaluated frozen S/4C₀ acoustic features on four-class IEMOCAP emotion recognition. A linear readout gives 71.55 ± 2.46% UAR (mean recall across classes), versus 57.73 ± 1.40% for random features. These are five speaker-disjoint session folds with development-only readout selection; ± denotes session-fold SD. Together, the experiments extend the speech-side evidence to emotion, translation and spoken QA.

3. Contribution and relation to ASR scaling.

Here, native means that the acoustic frontend, encoder, adapter and language decoder are jointly pretrained from random initialization. We study how size, capacity placement and speech/text exposure jointly affect both recognition and language modeling, including prediction on held-out configurations. The language evaluation covers 22 external tasks.

The suggested works are relevant: Li et al. study multilingual ASR capacity allocation; Zheng et al. (the cited Xiao et al. preprint) study large-scale ASR transfer; OWLS covers recognition and translation. Our additional question is the joint resource allocation among acoustic, decoder and embedding capacity, and between speech and standalone text supervision. We do not claim priority for the IsoFLOP principle itself.

4. Data and a reading guide to the comparisons.

The speech pool contains approximately 1.5 million hours of Internet speech–transcript data, predominantly Chinese and English; the text pool provides roughly one trillion supervised targets. Private corpus access limits replication of the original training, while the numerical supplement supports reproduction of the analyses.

To make the setup explicit here: Figure 2 varies model size at fixed compute; the allocation experiments redistribute capacity; the mixture experiments redistribute speech/text supervision. These are different controls. The fitted responses are validation NLLs, distinct from greedy AISHELL CER, LibriSpeech WER, and the 22-task text aggregate. The principal recognition results use no benchmark-specific fine-tuning; the new supervised transfer experiments above are reported separately.

Thank you for the clarification on Figure 2, that makes sense. This would have been easier to understand if 3.2 and 4.1 were written clearly, or refer to each other via a backpointer. And how was the B run (C0=3.6173×1019C_0 = 3.6173 × 10^{19} FLOPs) chosen as the IsoFLOPS reference point for the experiments? There seems to be no explanation for this, nor how the parameter sizes / model shapes are decided.

While the results on ST and SQA are nice to have, I struggle to see how they are useful. What is the point in comparing a single point of the scaling curve to random init? It still doesn't change how the paper is effectively another form of ASR scaling laws.

Ultimately, my biggest concern is the writing, which the authors did not appear to address. While the paper has an interesting idea, it is not human readable in its current form. So I choose to maintain my score of 1.

Thank you for suggesting the link between §§3.2 and 4.1. We are glad the Figure 2 clarification helped. We used abbreviations to keep repeated comparisons concise, but some passages became harder to follow.

What we changed in the revised manuscript.
Figure 1 now spells out automatic speech recognition (ASR) and negative log-likelihood (NLL). Section 3.3 distinguishes token loss from character and word error rates. Moving the benchmark example there lets §4.1 open with a single comparison.

Section 3.2 now explains why we chose the 463M model, B: its complete 125,000-update run gave us a reference budget, C0=3.6173×1019C_0 = 3.6173 \times 10^{19} FLOPs. Across six sizes we varied width and depth together while keeping roughly half the non-embedding parameters on each side. B has 14 encoder and 18 decoder layers but about 163M non-embedding parameters on each side. S fills the gap between T and B; XL roughly doubles L's non-embedding capacity. The new table gives all six shapes and update counts.

Section 4.1 refers back to §3.2: “At 1C01C_0, B trains for 125,000 updates and L for 42,730. Each L update costs more, so L sees fewer targets under the same speech/text recipe.” Figure 2 now distinguishes measured endpoints from fitted curves. Figure 3 distinguishes the acoustic share of non-embedding parameters from the transcript share of supervised tokens.

If you have a chance to look at these changes, does this approach make the comparisons easier to follow? If another sentence, transition or abbreviation still requires rereading, we would be grateful for one or two examples. That would help us focus the next revision.

What do the downstream comparisons test?

Comparing against random initialization under identical adaptation asks whether pretraining helps. On 512 answerable spoken QA examples, replacing the spoken question while holding the written passage fixed reduced the pretrained S/4C04C_0 model's answer F1 from 42.09 to 1.96. This checks that it uses the audio.

We adapted 20 checkpoints for English-to-Chinese translation on the same CoVoST 2 subset (original split IDs matched to Common Voice 15 audio; 15,522 test clips). All trained for three epochs, with checkpoint selection by development loss. Columns fix pretraining compute. Adaptation used equal data and epochs.

Translation BLEU (higher is better)0.5C00.5C_01C01C_02C02C_04C04C_0
T (149M)19.4121.3422.6924.58
S (327M)20.4824.8928.6229.77
B (463M)19.0125.1927.7229.60
M (794M)16.2921.0326.3329.74
L (1.45B)11.7513.6322.1829.01

BLEU rises with budget in every row. S leads L by over 11 BLEU at 1C01C_0 but by less than one at 4C04C_0.

What does joint speech–text training add to ASR scaling?

Earlier work examines ASR capacity, ASR transfer and recognition and translation. Standalone text is an explicit training objective: we fit its loss and evaluate 140 configurations on 22 external text task settings. Adding text can help recognition at fixed speech exposure (§4.4; extra compute). At 0.79B parameters and 0.5C00.5C_0, shifting capacity to the encoder lowered transcript loss (0.2855 to 0.2649) but raised text loss (2.4718 to 2.5930).

With M excluded from fitting, we used the law to rank 15 M designs at 0.75C00.75C_0 from their recorded exposures (§5.3). Minimizing predicted transcript loss favored an encoder-heavy split (81% acoustic). Suppose in some scenarios text is the priority and up to 3% more predicted transcript loss is acceptable. Minimizing predicted text loss then favored a balanced split (49% acoustic). Both used 30% transcript tokens. Measured transcript loss rose 2.2%, but text loss fell 5.1% and LAMBADA NLL fell from 3.67 to 3.37. This shows our law can serve more use other than ASR.

Thank you again for the helpful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.