acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Text-First Pretraining for Audio Language Models

Abstract

How should a fixed pretraining budget be allocated when extending a language model from text to audio? Large Audio Language Models (LALMs) are often built by starting from a pretrained text-only language model and then adding audio tokens through continued pretraining. This text-first strategy reuses an already capable linguistic backbone, but it also commits part of the total training budget to a text-only objective before the model begins joint text–audio learning. Determining whether this sequential allocation is more effective than learning both modalities from the beginning requires a controlled comparison. We therefore compare text-first continued pretraining (CPT) with joint text–audio pretraining from scratch (SPT) under matched end-to-end training budgets. We conduct a controlled sweep over training budgets, text–audio mixture ratios, and model scales while holding the decoder-only architecture and data sources fixed across CPT and SPT. Across the evaluated settings, SPT generally achieves a better balance between text and audio performance than CPT. Our results suggest that CPT can be inefficient because part of the budget is spent optimizing a text-only objective before training shifts to the joint text–audio objective, while text replay further reduces the budget available for audio-side training. SPT's advantage, however, does not appear to arise from simple positive cross-modal transfer, as the two objectives can still interfere. Instead, the results suggest that allowing both objectives to shape the model throughout training contributes to a better balance between text and audio capabilities. This trend extends to a Mixture-of-Experts architecture and a speech-generation objective, and persists after supervised fine-tuning on speech question answering.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large audio language models, multimodal pretraining, training order, continued pretraining, discrete audio tokens

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dwaUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares text-first continued pretraining (CPT) with joint text-audio pretraining from scratch (SPT), accounting for text-backbone pretraining within a fixed total token budget. With matched cumulative text and audio token allocations, SPT generally achieves a better text-audio performance trade-off across the evaluated settings. Additional experiments examine transition-specific optimization, model scale, MoE architectures, speech generation, and downstream speech question answering.

Critical strengths
  1. The budget accounting is well motivated. Matching both total training tokens and cumulative modality allocations provides a useful comparison of training curricula and avoids treating text-backbone pretraining as a free resource.

  2. The paper examines different budgets, mixture ratios, model sizes, and optimization settings, with additional multi-seed experiments. The learning-rate ablation is useful because it reveals substantial sensitivity in the CPT baseline.

Critical weaknesses
  1. Figure 2 shows that transition-specific optimization substantially narrows the CPT-SPT gap. However, the multi-seed results in Table 3 only evaluate the default CPT setup. It therefore remains unclear whether the smaller gap between SPT and tuned CPT is consistent across random seeds. Reporting multi-seed results for the competitive CPT variants would strengthen the central conclusion.

  2. The comparison does not explore when audio should be introduced under a fixed budget. The main matched-budget experiments devote half of CPT's total training tokens to text-only pretraining. It remains unclear whether a shorter text-only stage, followed by a longer mixed stage with the same cumulative text/audio token counts, could close the gap with SPT.

  3. The study primarily establishes a recipe preference for small-scale, discrete-token English speech models. Its extensions add breadth but do not resolve the baseline and curriculum-selection concerns. Moreover, ASR and transcription-based TTS evaluation provide limited evidence about broader audio understanding or speech generation quality.

Questions for authors

NA

5 comments

We thank the reviewer for the careful review and constructive feedback, which helped us strengthen the robustness of our experimental evaluation.


1. Multi-Seed Results Across Experimental Settings (W1)

We compared SPT with transition-tuned CPT across three seeds. In the 60B-token setting, SPT achieves higher mean text and audio performance than Default CPT, LR-Matched CPT, and Low-LR CPT at all five allocations. Transition tuning improves CPT performance and stability, but SPT retains its advantage in mean performance. The overall trend against Default CPT also holds across the other training budgets and model scales.

The original Table 3 evaluated only Default CPT. We added two CPT transition strategies in the 60B-token setting and extended the 200B-token and smaller-model comparisons in Figure 1 to three seeds each. All values are mean ± standard deviation. Bold values indicate the highest mean performance for each modality at each allocation.

60B-token setting

Default CPTLR-Matched CPTLow-LR CPTSPT
Text:AudioTextAudioTextAudioTextAudioTextAudio
39:2143.07 ± 0.3848.60 ± 10.8343.87 ± 0.4282.90 ± 2.3543.60 ± 0.2082.80 ± 1.1546.05 ± 0.4985.50 ± 0.42
42:1844.00 ± 0.1776.77 ± 8.0744.77 ± 0.2383.20 ± 2.0843.97 ± 0.2177.50 ± 9.2145.23 ± 0.2584.63 ± 1.00
45:1544.70 ± 0.5077.83 ± 6.4845.67 ± 0.3583.70 ± 0.8244.47 ± 0.1570.77 ± 3.0245.77 ± 0.6184.43 ± 0.32
48:1245.13 ± 0.2162.97 ± 29.9245.87 ± 0.1580.77 ± 1.1944.97 ± 0.5573.30 ± 11.0346.10 ± 0.2081.97 ± 0.42
51:945.57 ± 0.0643.53 ± 19.4046.17 ± 0.2579.67 ± 1.7045.23 ± 0.4567.50 ± 4.8846.57 ± 0.3580.80 ± 0.61

200B-token setting

Text:AudioDefault CPT TextDefault CPT AudioSPT TextSPT Audio
130:7045.33 ± 0.2583.77 ± 1.9648.30 ± 0.6085.90 ± 1.15
140:6046.60 ± 0.6274.90 ± 8.0948.37 ± 0.3285.40 ± 0.70
150:5047.30 ± 0.4675.37 ± 8.8948.73 ± 0.4283.93 ± 2.20
160:4048.33 ± 0.3557.57 ± 12.7649.80 ± 0.2681.57 ± 2.51
170:3048.70 ± 0.3670.10 ± 6.3249.60 ± 0.3082.97 ± 0.93

Smaller-model setting

Text:AudioDefault CPT TextDefault CPT AudioSPT TextSPT Audio
65:3539.93 ± 0.4771.37 ± 9.3041.53 ± 0.8182.10 ± 1.20
70:3040.67 ± 0.2575.87 ± 3.1842.43 ± 0.0680.67 ± 2.75
75:2540.87 ± 0.2969.00 ± 8.9441.90 ± 0.2877.30 ± 0.28
80:2041.60 ± 0.1076.83 ± 3.9042.90 ± 0.6175.70 ± 6.81
85:1541.43 ± 0.2957.73 ± 16.2543.25 ± 0.3571.05 ± 3.61

2. Sensitivity to the Text-Only Pretraining Duration (W2)

To test whether a shorter text-only stage could close the gap with SPT, we reduced text-only pretraining from 30B to 15B tokens and extended mixed CPT from 30B to 45B tokens, keeping the total 60B-token budget and cumulative text–audio allocations fixed. This improves CPT’s mean text and audio performance and reduces variation in audio performance across seeds at every allocation. SPT still achieves higher mean text performance at all five allocations and higher mean audio performance at three, with trade-offs at the remaining two. Thus, earlier audio introduction strengthens CPT, but SPT remains generally competitive.

The original 30B-token text-only budget followed the Chinchilla-style token-to-parameter ratio, with an equal budget assigned to mixed CPT. The resulting 50% split was not selected as the optimal switch point.

Both CPT settings use the Default CPT transition strategy. Below, 30B and 15B denote the text-only pretraining budgets. All results are mean ± standard deviation across three seeds.

Text:Audio30B CPT Text30B CPT Audio15B CPT Text15B CPT AudioSPT TextSPT Audio
39:2143.07 ± 0.3848.60 ± 10.8344.73 ± 0.5785.60 ± 0.3646.05 ± 0.4985.50 ± 0.42
42:1844.00 ± 0.1776.77 ± 8.0744.50 ± 0.2884.05 ± 1.2045.23 ± 0.2584.63 ± 1.00
45:1544.70 ± 0.5077.83 ± 6.4845.07 ± 0.6783.10 ± 0.6145.77 ± 0.6184.43 ± 0.32
48:1245.13 ± 0.2162.97 ± 29.9245.45 ± 0.0782.70 ± 0.7146.10 ± 0.2081.97 ± 0.42
51:945.57 ± 0.0643.53 ± 19.4045.75 ± 0.2178.60 ± 1.5646.57 ± 0.3580.80 ± 0.61

3. Scope of Generalization and Audio Evaluation (W3)

We agree that ASR and transcription-based TTS evaluation alone cannot establish broad audio understanding or overall speech generation quality. Similar to Moshi [1] and Qwen2.5-Omni [2], our TTS extension uses WER computed from transcriptions of generated speech to evaluate how accurately it conveys the target text. The purpose is to test whether differences between training strategies also appear in content accuracy, rather than to evaluate naturalness, audio quality, or speaker similarity. We will clarify this purpose and evaluation scope in the paper.

Reference

  • [1] Moshi: a speech-text foundation model for real-time dialogue, 2024
  • [2] Qwen2.5-Omni Technical Report, 2025

We thank the reviewer for the constructive feedback, which has helped us clarify our claims and strengthen the empirical evidence.

Reviewer_rESZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares text-first continued pretraining (CPT) with joint text–audio pretraining from scratch (SPT), matching total tokens and cumulative modality allocations. SPT generally offers a better text–audio trade-off. Experiments cover training budgets, model scales, transition schedules, MoE, speech generation, and speech QA.

Critical strengths

Matching cumulative text/audio exposure, including backbone pretraining, makes training order meaningfully testable (4.1). This is stronger evidence than comparing equal adaptation budgets alone. Transition ablations, three-seed results at 60B tokens, and MoE/TTS/QA extensions support the empirical trend (4.3–4.4, 5). The paper also correctly separates this setting from reusing an available pretrained checkpoint.

Critical weaknesses

The strongest comparator is not fully characterized. Transition tuning substantially improves CPT and sometimes beats SPT (Fig. 2), but Table 3 does not establish multi-seed robustness for each tuned CPT variant. The remaining frontier advantage needs uncertainty estimates. Generality remains limited to one tokenizer and relatively small models; the TTS extension measures intelligibility through WER alone (3.2, 5.2). Thus, the evidence supports this training regime more strongly than broad audio-model quality.

Questions for authors

Clarification: Which CPT schedule is used in Table 3? Does SPT retain its advantage against the best validation-selected transition schedule across seeds? Clarification: How was the text-only stage length selected? Would varying that split at fixed total and modality-token budgets materially change Fig. 2?

3 comments

We thank the reviewer for the careful review and constructive feedback, which helped us strengthen the robustness of our experimental evaluation.


1. Multi-Seed Results Across Experimental Settings (W1, Q1)

The CPT results in the original Table 3 use Default CPT. We extended the comparisons to three seeds in each setting. In the 60B-token setting, SPT achieves higher mean text and audio performance than Default CPT, LR-Matched CPT, and Low-LR CPT at all five allocations. In the 200B-token setting, SPT also achieves higher mean performance in both modalities than Default CPT at every allocation. For the smaller model, SPT achieves higher mean text performance at all five allocations and higher mean audio performance at four of them.

Rather than selecting one transition strategy on a validation set, we report all three evaluated CPT strategies. In the 60B-token setting, SPT retains its advantage in mean performance in both modalities at every matched allocation, regardless of which reported CPT strategy is used for comparison.

All results below are mean ± standard deviation across three seeds. Bold values indicate the highest mean performance for each modality at each allocation.

60B-token setting

Default CPTLR-Matched CPTLow-LR CPTSPT
Text:AudioTextAudioTextAudioTextAudioTextAudio
39:2143.07 ± 0.3848.60 ± 10.8343.87 ± 0.4282.90 ± 2.3543.60 ± 0.2082.80 ± 1.1546.05 ± 0.4985.50 ± 0.42
42:1844.00 ± 0.1776.77 ± 8.0744.77 ± 0.2383.20 ± 2.0843.97 ± 0.2177.50 ± 9.2145.23 ± 0.2584.63 ± 1.00
45:1544.70 ± 0.5077.83 ± 6.4845.67 ± 0.3583.70 ± 0.8244.47 ± 0.1570.77 ± 3.0245.77 ± 0.6184.43 ± 0.32
48:1245.13 ± 0.2162.97 ± 29.9245.87 ± 0.1580.77 ± 1.1944.97 ± 0.5573.30 ± 11.0346.10 ± 0.2081.97 ± 0.42
51:945.57 ± 0.0643.53 ± 19.4046.17 ± 0.2579.67 ± 1.7045.23 ± 0.4567.50 ± 4.8846.57 ± 0.3580.80 ± 0.61

200B-token setting

Text:AudioDefault CPT TextDefault CPT AudioSPT TextSPT Audio
130:7045.33 ± 0.2583.77 ± 1.9648.30 ± 0.6085.90 ± 1.15
140:6046.60 ± 0.6274.90 ± 8.0948.37 ± 0.3285.40 ± 0.70
150:5047.30 ± 0.4675.37 ± 8.8948.73 ± 0.4283.93 ± 2.20
160:4048.33 ± 0.3557.57 ± 12.7649.80 ± 0.2681.57 ± 2.51
170:3048.70 ± 0.3670.10 ± 6.3249.60 ± 0.3082.97 ± 0.93

Smaller-model setting

Text:AudioDefault CPT TextDefault CPT AudioSPT TextSPT Audio
65:3539.93 ± 0.4771.37 ± 9.3041.53 ± 0.8182.10 ± 1.20
70:3040.67 ± 0.2575.87 ± 3.1842.43 ± 0.0680.67 ± 2.75
75:2540.87 ± 0.2969.00 ± 8.9441.90 ± 0.2877.30 ± 0.28
80:2041.60 ± 0.1076.83 ± 3.9042.90 ± 0.6175.70 ± 6.81
85:1541.43 ± 0.2957.73 ± 16.2543.25 ± 0.3571.05 ± 3.61

2. Sensitivity to the Text-Only Pretraining Duration (Q2)

We added a shorter text-only pretraining setting, reducing this stage from 30B to 15B tokens and extending mixed CPT from 30B to 45B tokens while keeping the total 60B-token budget and cumulative modality allocations fixed. CPT’s mean text and audio performance and audio stability improve at every allocation. SPT still achieves higher mean text performance at all five allocations and higher mean audio performance at three. Thus, the split affects the gap, but the results generally favor earlier joint text–audio training.

The original 30B-token text-only budget followed the Chinchilla-style token-to-parameter ratio, with an equal budget assigned to mixed CPT. The resulting 50% split was not selected as optimal.

Both CPT settings use the Default CPT transition strategy. Below, 30B and 15B denote the text-only pretraining budgets. All results are mean ± standard deviation across three seeds.

Text:Audio30B CPT Text30B CPT Audio15B CPT Text15B CPT AudioSPT TextSPT Audio
39:2143.07 ± 0.3848.60 ± 10.8344.73 ± 0.5785.60 ± 0.3646.05 ± 0.4985.50 ± 0.42
42:1844.00 ± 0.1776.77 ± 8.0744.50 ± 0.2884.05 ± 1.2045.23 ± 0.2584.63 ± 1.00
45:1544.70 ± 0.5077.83 ± 6.4845.07 ± 0.6783.10 ± 0.6145.77 ± 0.6184.43 ± 0.32
48:1245.13 ± 0.2162.97 ± 29.9245.45 ± 0.0782.70 ± 0.7146.10 ± 0.2081.97 ± 0.42
51:945.57 ± 0.0643.53 ± 19.4045.75 ± 0.2178.60 ± 1.5646.57 ± 0.3580.80 ± 0.61

3. Generality Across Tokenizers and Model Scales (W1)

We observed similar trends in early experiments with the CosyVoice 2 tokenizer. After CosyVoice 3 was released, we used it for the full set of experiments. We also compared 0.5B and 1.5B models and Dense and MoE architectures, examining the effect of training order across these settings. We acknowledge that broader generalization across tokenizers and model scales requires further validation. Given the substantial computational cost, we leave this to future work.

4. Scope of Generalization and Audio Evaluation (W1)

We agree that WER alone cannot establish overall TTS quality, including naturalness, audio quality, and speaker similarity. WER computed from transcriptions of generated speech is also used in Moshi [1] and Qwen2.5-Omni [2] to measure how accurately the generated speech conveys the target text. Our TTS extension tests whether differences between training strategies also appear in this aspect of speech generation. We will clarify this purpose and evaluation scope in the paper.

Reference

  • [1] Moshi: a speech-text foundation model for real-time dialogue, 2024
  • [2] Qwen2.5-Omni Technical Report, 2025

We thank the reviewer for the constructive feedback, which has helped us clarify our claims and strengthen the empirical evidence.

Reviewer_wKMzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares text-first continued pretraining (CPT) with joint text–audio pretraining from scratch (SPT), including text-backbone pretraining in the total token budget. The main comparisons match cumulative text-only and paired speech–transcript token allocations while varying training budgets, mixture ratios, and model sizes. SPT generally achieves a better text–ASR trade-off, although transition-specific optimization substantially improves CPT. Additional experiments examine cross-modal interference, backbone initialization, an MoE architecture, speech generation, and speech question answering after supervised fine-tuning. Local curvature and logit-lens analyses provide diagnostic evidence for differences between the training strategies. The central contribution is a controlled empirical assessment of pretraining order and its implications for constructing discrete-token audio language models.

Critical strengths
  • Including text-backbone pretraining and matching cumulative text and paired-ASR token allocations avoids attributing an advantage to additional training data or an unaccounted-for initialization. The budget, mixture, and model-size sweeps, together with the three-seed results in the 60B-token setting, provide evidence that the observed difference is not restricted to a single allocation or initialization. This is the strongest empirical contribution.

  • The paper investigates alternative explanations and acknowledges qualifications to its conclusion. Section 4.3 shows that transition-specific optimization can substantially improve CPT, while Section 4.7 shows that SPT’s advantage does not coincide with positive cross-modal transfer in the tested settings. These experiments add information beyond a simple comparison of final scores. The authors also appropriately present curvature and representation analyses as diagnostics rather than established causal explanations.

Critical weaknesses
  • The explanatory and practical contribution remains limited relative to the breadth of the experiments. The study establishes a preferred recipe in the evaluated settings, but provides limited characterization of when this preference changes or how a practitioner should choose a backbone, modality-introduction schedule, or replay allocation. The practical recommendation largely distinguishes whether backbone pretraining must be paid for; it offers little guidance within either scenario. The curvature and logit-lens analyses are appropriately qualified, but their usefulness for predicting adaptation outcomes or selecting a training strategy is not validated. Moreover, text replay consuming budget cannot by itself explain the matched-allocation performance gap, since SPT processes the same cumulative text allocation. Consequently, the paper provides a useful observation, but less explanatory or actionable insight than its practical-guidance contribution suggests.

  • The robustness of the advantage over competitive CPT configurations is not established as convincingly as the default comparison. Figure 2 shows that transition tuning substantially narrows the gap and that CPT can outperform SPT at some allocations. However, Table 3 does not provide comparable multi-seed results for these tuned variants, despite the large seed variation observed for CPT. Table 4 likewise does not establish whether the downstream advantage persists against the tuned CPT alternatives. In addition, the main matched-budget comparisons allocate half of the total budget to text-only pretraining, leaving the sensitivity to this substantial text-only prefix unresolved. These limitations do not invalidate the reported comparisons, but they reduce confidence in how broadly the recommendation extends beyond the particular sequential schedules evaluated.

Questions for authors
  • Could you provide numerical results underlying Figure 2 and clarify what “better overall balance” means when comparing SPT against the combined set of transition-tuned CPT configurations? In particular, which allocations remain dominated by SPT, and where do the strategies offer different trade-offs?

  • Does the remaining SPT advantage persist across seeds for one or two representative allocations using the strongest transition-tuned CPT configuration? Given the variance in Table 3, this would materially affect my confidence in the recommendation. Please also clarify which CPT transition strategy was used for Table 4.

  • What motivates allocating half of the total budget to text-only pretraining in the main comparisons? Is there evidence from existing runs or a representative alternative switch point that the conclusion persists with a shorter text-only stage, while keeping cumulative allocations matched? If not, how narrowly should the recommendation be interpreted?

5 comments

We thank the reviewer for the careful review and constructive feedback, which helped us strengthen the robustness of our experimental evaluation.


1. Multi-Seed Results Across Experimental Settings (W2, Q1, Q2)

After extending the comparisons to three random seeds each, SPT generally retains its advantage in mean performance against transition-tuned CPT and across different training budgets and model scales. In the 60B-token setting, SPT achieves higher mean text and audio performance than all CPT variants at all five allocations. In the 200B-token setting, SPT also achieves higher mean performance in both modalities than Default CPT at every allocation. For the smaller model, SPT achieves higher mean text performance at all five allocations and higher mean audio performance at four of them.

To address the request for the numerical results underlying Figure 2 and the robustness of transition-tuned CPT across seeds, we report the 60B-token comparison below as mean ± standard deviation across three seeds. We compare Default CPT, LR-Matched CPT, Low-LR CPT, and SPT at each allocation. Bold values indicate the highest mean performance for each modality at each allocation.

60B-token setting: Comparison of CPT transition strategies

Default CPTLR-Matched CPTLow-LR CPTSPT
Text:AudioTextAudioTextAudioTextAudioTextAudio
39:2143.07 ± 0.3848.60 ± 10.8343.87 ± 0.4282.90 ± 2.3543.60 ± 0.2082.80 ± 1.1546.05 ± 0.4985.50 ± 0.42
42:1844.00 ± 0.1776.77 ± 8.0744.77 ± 0.2383.20 ± 2.0843.97 ± 0.2177.50 ± 9.2145.23 ± 0.2584.63 ± 1.00
45:1544.70 ± 0.5077.83 ± 6.4845.67 ± 0.3583.70 ± 0.8244.47 ± 0.1570.77 ± 3.0245.77 ± 0.6184.43 ± 0.32
48:1245.13 ± 0.2162.97 ± 29.9245.87 ± 0.1580.77 ± 1.1944.97 ± 0.5573.30 ± 11.0346.10 ± 0.2081.97 ± 0.42
51:945.57 ± 0.0643.53 ± 19.4046.17 ± 0.2579.67 ± 1.7045.23 ± 0.4567.50 ± 4.8846.57 ± 0.3580.80 ± 0.61

LR-Matched CPT achieves higher mean text and audio performance than Default CPT at every allocation, with a smaller standard deviation in audio performance across seeds. This supports the reviewer’s point that the learning-rate setting at the CPT transition affects performance and stability. Even after these improvements, however, SPT retains higher mean performance in both modalities at the same allocation.

We also extend the Default CPT–SPT comparisons in the 200B-token and smaller-model settings of Figure 1 to three seeds each.

200B-token setting

Text:AudioDefault CPT TextDefault CPT AudioSPT TextSPT Audio
130:7045.33 ± 0.2583.77 ± 1.9648.30 ± 0.6085.90 ± 1.15
140:6046.60 ± 0.6274.90 ± 8.0948.37 ± 0.3285.40 ± 0.70
150:5047.30 ± 0.4675.37 ± 8.8948.73 ± 0.4283.93 ± 2.20
160:4048.33 ± 0.3557.57 ± 12.7649.80 ± 0.2681.57 ± 2.51
170:3048.70 ± 0.3670.10 ± 6.3249.60 ± 0.3082.97 ± 0.93

Smaller-model setting

Text:AudioDefault CPT TextDefault CPT AudioSPT TextSPT Audio
65:3539.93 ± 0.4771.37 ± 9.3041.53 ± 0.8182.10 ± 1.20
70:3040.67 ± 0.2575.87 ± 3.1842.43 ± 0.0680.67 ± 2.75
75:2540.87 ± 0.2969.00 ± 8.9441.90 ± 0.2877.30 ± 0.28
80:2041.60 ± 0.1076.83 ± 3.9042.90 ± 0.6175.70 ± 6.81
85:1541.43 ± 0.2957.73 ± 16.2543.25 ± 0.3571.05 ± 3.61

Overall, these additional multi-seed results show that SPT generally maintains its advantage in mean performance and provides a better text–audio balance across the evaluated training budgets and model scales.

In the revised paper, Figure 2 reports three-seed mean results for the 60B-token setting; the original single-seed 200B-token transition comparison is retained in Appendix F (Figure 9).

2. Sensitivity to the Text-Only Pretraining Duration (W2, Q3)

Keeping the total 60B-token budget and final cumulative text–audio token allocations fixed, we shortened text-only pretraining from 30B to 15B tokens and extended the mixed CPT stage from 30B to 45B tokens. This improved CPT’s mean text and audio performance and reduced variation in audio performance across seeds at every allocation. SPT achieves higher mean text performance than 15B CPT at all five allocations and higher mean audio performance at three of them. These results suggest that, under the evaluated conditions, starting joint training earlier is generally more favorable than allocating more of the budget to text-backbone pretraining.

In the original experiments, we set the text-backbone pretraining budget based on the Chinchilla-style token-to-parameter ratio and assigned the same token budget to the subsequent CPT stage. This resulted in a text-only stage covering 50% of the total budget; we did not select this split as the optimal switch point. The additional 15B-token setting falls below this reference budget and therefore starts audio adaptation from a relatively undertrained text backbone.

Both CPT settings use the Default CPT transition strategy. Below, 30B and 15B denote the text-only pretraining budgets, and Text:Audio denotes the final cumulative token allocation. All results are reported as mean ± standard deviation across three seeds.

Text:Audio30B CPT Text30B CPT Audio15B CPT Text15B CPT AudioSPT TextSPT Audio
39:2143.07 ± 0.3848.60 ± 10.8344.73 ± 0.5785.60 ± 0.3646.05 ± 0.4985.50 ± 0.42
42:1844.00 ± 0.1776.77 ± 8.0744.50 ± 0.2884.05 ± 1.2045.23 ± 0.2584.63 ± 1.00
45:1544.70 ± 0.5077.83 ± 6.4845.07 ± 0.6783.10 ± 0.6145.77 ± 0.6184.43 ± 0.32
48:1245.13 ± 0.2162.97 ± 29.9245.45 ± 0.0782.70 ± 0.7146.10 ± 0.2081.97 ± 0.42
51:945.57 ± 0.0643.53 ± 19.4045.75 ± 0.2178.60 ± 1.5646.57 ± 0.3580.80 ± 0.61

Shortening the text-only stage improves CPT, but SPT retains higher mean text performance at all allocations and higher mean audio performance at three of five. Overall, these results favor earlier joint text–audio training over allocating more of the fixed budget to text-only pretraining.

3. Downstream Evaluation Against Transition-Tuned CPT (W2, Q2)

The original Table 4 (Table 5 in the revised paper) used Default CPT. To address the concern about transition-tuned CPT, we added results for LR-Matched CPT before and after LibriSQA fine-tuning, using the same 200B-token budget and final cumulative 17:3 text–audio allocation.

SPT generally retains its downstream advantage after fine-tuning, scoring higher than LR-Matched CPT on five of seven evaluations and tying on one. In particular, SPT achieves 7.6 and 4.9 percentage points higher EM on LibriSQA P2 and DREAM-TTS, respectively, while LR-Matched CPT scores 0.2 points higher on PublicSG. Below, D-CPT and LM-CPT denote Default CPT and LR-Matched CPT, respectively, and FT denotes results after LibriSQA fine-tuning.

TaskDatasetD-CPTLM-CPTSPTD-CPT-FTLM-CPT-FTSPT-FT
P1 (ROUGE-L)LibriSQA P129.337.029.056.656.757.2
P1PublicSG18.923.420.536.237.537.3
P1SLUE_SQA52.94.29.814.715.816.1
P1SpokenSQuAD3.33.45.411.211.411.4
P2 (EM)LibriSQA P20.10.00.157.659.967.5
P2CN Colleage0.00.00.135.535.435.7
P2DREAM-TTS0.00.10.236.135.440.3

4. Clarifying the Explanation and the Role of Diagnostic Analyses (W1)

Our discussion of replay concerns the potential inefficiency of allocating part of the CPT-stage budget to preserving previously acquired text capability, whereas SPT develops text and audio capabilities jointly from the beginning. Under matched cumulative allocations, CPT and SPT process the same total numbers of text and audio tokens, so replay consuming budget cannot, by itself, explain the observed performance gap. The relevant distinction is when the objectives shape the model: CPT first learns under a text-only objective and subsequently integrates audio while preserving text capability, whereas SPT learns under both objectives from the beginning. This motivates a training-dynamics interpretation of the results, without establishing a causal explanation for the gap. We have clarified this distinction in the paper.

The curvature analysis is a diagnostic analysis of local adaptation behavior. It has not been validated as a tool for predicting adaptation outcomes or selecting an optimal training strategy. Likewise, the logit-lens analysis in the Appendix is a supporting analysis of differences in the internal behavior of CPT and SPT, rather than a test of which strategy is better. We have clarified the purpose and scope of each analysis.

5. Practical Guidance and Scope of the Recommendations (W1, Q3)

Our practical recommendation depends on whether a suitable pretrained text backbone is already available. When a released checkpoint matches the desired architecture, model scale, and tokenizer, CPT is a practical choice because its pretraining cost does not need to be paid again. When the backbone must be trained from scratch, our matched-budget results support considering SPT first, rather than treating text-only pretraining as a required stage.

The additional experiments also provide guidance for configuring CPT. In the evaluated 60B-token setting, LR-Matched CPT improves mean text and audio performance and reduces audio variability relative to Default CPT at every allocation. Shortening the text-only stage from 30B to 15B tokens likewise improves both mean scores at every allocation while keeping total and cumulative modality-token budgets fixed. These findings motivate maintaining a continuous learning-rate schedule at the transition and considering earlier audio introduction when designing a CPT recipe.

The shorter-prefix results also clarify where the choice depends on the target capability: SPT achieves higher mean text performance at all five allocations and higher mean audio performance at three, while shorter-prefix CPT offers higher mean audio performance at the other two. Thus, our recommendation is to consider SPT first when training the backbone is part of the budget, and to evaluate CPT configurations against the desired text–audio trade-off. We have incorporated these concrete implications into the practical discussion in the paper.

6. Clarifying “Better Overall Balance” (Q1)

“Better overall balance” means that SPT results generally lie toward the upper right of the text–audio performance plane, providing more favorable performance combinations. It does not mean that SPT is better in both modalities at every allocation. Specifically, in the 60B-token multi-seed comparison, SPT achieves higher mean text and audio scores than Default CPT, LR-Matched CPT, and Low-LR CPT at all five matched allocations. Thus, in this setting, SPT retains its advantage in mean performance for both modalities across all evaluated CPT transition strategies.


We thank the reviewer for the constructive feedback, which has helped us clarify our claims and strengthen the empirical evidence.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions and Responses

Dear Reviewers and AC,

We thank the reviewers for their careful reviews and constructive feedback, which helped us strengthen the robustness of our experimental evaluation. We summarize our main responses below.

  • Multi-seed robustness (wKMz, rESZ, dwaU). We compared all three CPT transition strategies and SPT across three seeds in the 60B-token setting. SPT achieves higher mean text and audio performance at all five matched allocations. Additional three-seed comparisons at 200B tokens and a smaller model scale generally maintain the trend against Default CPT.

  • Text-only stage duration (wKMz, rESZ, dwaU). We shortened text-only pretraining from 30B to 15B tokens while keeping the total budget and cumulative modality allocations fixed. CPT improves, but SPT retains higher mean text performance at all five allocations and higher mean audio performance at three.

  • Downstream evaluation (wKMz). We added LR-Matched CPT results before and after fine-tuning. After fine-tuning, SPT scores higher on five of seven evaluations and ties on one, with EM gains of 7.6 and 4.9 percentage points on LibriSQA P2 and DREAM-TTS.

  • Explanation and practical guidance (wKMz). We clarify that replay alone does not explain the matched-allocation gap and that our diagnostic analyses are not validated predictors. Our recommendation favors considering SPT when backbone training must be paid for, while CPT remains practical when a suitable released backbone is available.

  • Evaluation scope (rESZ, dwaU). We acknowledge the limits of generalization across tokenizers and model scales and clarify that TTS WER measures content accuracy rather than overall speech generation quality.

We have submitted a revised PDF incorporating these additional results and clarifications, with revisions marked in blue.

We thank the reviewers for their time and constructive feedback, which helped us improve the clarity of the paper, strengthen the robustness of our evaluation, and better define the scope of our conclusions.

AuthorsOpenReview ↗

Follow-up on Author Responses and Revised Manuscript

Dear Reviewers and AC,

As the discussion period is approaching its end, we would like to kindly follow up on our responses and revised manuscript. We have added substantial new experiments addressing the main concerns raised in the reviews, including multi-seed comparisons with transition-tuned CPT, sensitivity to the text-only pretraining duration, and downstream evaluation against LR-Matched CPT.

We hope these results address the key concerns and would appreciate the reviewers reconsidering their assessments in light of the new evidence. We would also be happy to clarify any remaining questions.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.