acceptodds
Under review as a conference paper at ICLR 2027

GSLM2: Multi-Stream Speech Representation Learning and Generative Spoken Language Modeling

42 views

Abstract

Generative spoken language models (SLMs) offer a route to learning linguistic structure purely from speech, yet they lag far behind text-based models in both capability and data scaling efficiency. We identify two bottlenecks underlying this gap: the limited linguistic quality of existing speech units and the reliance on a single stream of discrete units to represent speech. Multiple codebooks are widely used to encode acoustic features but rarely to represent speech semantics. We present GSLM2, which learns and models complementary properties of speech at multiple levels of abstraction. Its encoder, SpidR 2.0, uses self-supervised learning with differentiable prediction heads to induce high-quality discrete units at multiple levels. A multi-stream SLM then jointly models these units using a shared temporal backbone, stream-specific predictors, and an autoregressive head that captures cross-stream dependencies. Across training data sizes and model scales, up to 6M hours and 6.5B parameters, GSLM2 consistently improves linguistic competence while scaling more efficiently with data. As a result, it substantially narrows the gap between spoken and text-based LMs; across read and conversational speech domains, we shrink the gap by about two orders of magnitude on sWuggy, sBLIMP, and tSC, while also outperforming competitive speech-text systems on all three tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Speech, Textless NLP, Generative Spoken Language Modeling, Self-Supervised Learning, Speech Representations

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_PwrkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to close the large performance and data-efficiency gap between textless Spoken Language Models (SLMs) and Text LMs. The authors point out two main bottlenecks in current SLMs:

  1. Discrete speech units are noisy, unstable across random seeds, and require a separate post-hoc kk-means step.
  2. Standard SLMs only use units from a single encoder layer, forcing a trade-off between fine phonetic detail and word-level meaning.

To fix these issues, the authors propose GSLM2, which has two main parts:

  • SpidR 2.0 (Speech Tokenizer): A self-supervised student-teacher encoder that learns discrete units directly during training using shared, normalized prediction heads and entropy regularization (ME-MAX). This removes kk-means and creates stable, longer (~70–90 ms) phone-like units across multiple layers.
  • Multi-Stream SLM: A language model that jointly processes 6 unit streams (from layers 5 to 10 of SpidR 2.0) after removing repeated frames across streams. It uses a shared Transformer backbone with two types of heads:
    1. Stream-specific linear heads that predict when a stream changes to a new unit (trained with a self-distillation loss from a single-stream pass).
    2. An autoregressive depth Transformer head that generates the 6-stream unit tuple at each step.

The authors test GSLM2 on Libri-Light (up to 60K hours / 1.2B parameters), a 6M-hour conversational speech dataset (up to 6.5B parameters), and multilingual VoxPopuli data (English, French, German). GSLM2 beats single-stream baselines and larger speech/speech-text models (such as Moshi and Spirit-LM), cutting the data-scaling gap to Text LMs from 4 orders of magnitude down to 2.

Critical strengths
  1. Cleaner, More Stable Speech Units Without kk-Means (SpidR 2.0): Removing post-hoc kk-means and learning normalized codebooks directly during pretraining works very well. It cuts cross-seed ABX variance by 10×10\times, reaches a strong 2.83% triphone ABX error, and keeps high quality across layers 5 to 12 instead of peaking at just one layer (Figure 3). It also naturally compresses speech into longer (~70–90 ms) units at roughly half the bitrate, which makes language modeling much easier.

  2. Effective Multi-Stream Modeling and Self-Distillation: Combining 6 layers consistently beats using any single layer. The authors include great control tests (Table 13) proving that simply repeating or sampling from the best layer (Layer 7) six times does not give the same boost. They also show through ablations that their self-distillation loss (Ldistill\mathcal{L}_{\text{distill}})—which aligns multi-stream predictions with a single-stream pass—is the key ingredient, adding +7.8 to +9.9 accuracy points (Table 11).

  3. Thorough Scaling Study and Useful New Benchmarks: The paper scales up to 6M hours of real conversational speech and 6.5B parameters, with fair Text LM and cascaded ASR+LM baselines. It also introduces valuable new benchmarks (Spoken LongTail-Swap and French/German evaluations), multi-seed variance breakdowns, and a 20-encoder study showing that Word MAP predicts downstream SLM performance much better than ABX or PNMI.

Critical weaknesses
  1. "Different Levels of Abstraction" Looks More Like a Same-Scale Phone Ensemble: The paper claims that the 6 streams capture different levels of linguistic abstraction (from fine phonetics to high-level semantics). However, the appendix data tells a simpler story:

    • Across layers 5–10 of SpidR 2.0, unit duration (70–80 ms), bitrate (90–100 bps), and phone purity (~60%) are almost flat (Figure 12).
    • Pairwise overlap between layers is very high (mean NMI of 0.71 in Figure 18, up from 0.55 in original SpidR).
    • Counter-intuitively, Phoneme ABX actually peaks in a deeper layer (Layer 10) than Word MAP (Layer 7) (Table 5).
    • In Table 12, the 6 individual stream heads inside GSLM2 get very similar scores on their own. Averaging them helps word and syntax scores (sWuggy +3.5%, sBLIMP +2.0%), but does not beat Layer 9 alone on semantics (tSC: 78.9% vs. 79.3%).
    • Together, this suggests GSLM2 is not learning a true coarse-to-fine hierarchy (like phones →\rightarrow syllables →\rightarrow words). Instead, it works like a multi-view ensemble at the same ~70 ms phone scale, where slightly different unit boundaries across layers regularize the shared backbone.
  2. ~3×3\times Compute Cost for a Mostly Constant Scaling Shift: Because cross-stream deduplication keeps a frame whenever any of the 6 streams changes, sequences are about 2×2\times longer than in single-stream SpidR 2.0. Combined with the extra single-stream pass needed for self-distillation, GSLM2 uses 3×3\times to 3.8×3.8\times more training FLOPs and 2.4×2.4\times to 3.4×3.4\times more inference FLOPs per second (Table 10). Meanwhile, the scaling curves (Figure 6) show that upgrading the tokenizer to SpidR 2.0 drives most of the scaling slope improvement (Δγ=+0.016\Delta\gamma = +0.016), while 6-stream modeling mostly gives a constant vertical shift (Δγ=+0.005\Delta\gamma = +0.005).

Questions for authors

see weaknesses

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Response to Reviewer Pwrk

We appreciate the reviewer's time and effort in providing a comprehensive review and raising thoughtful questions, which we address point by point below. The corresponding revisions, including additional analyses, are highlighted in blue in the revised manuscript.

W1

"Different Levels of Abstraction" Looks More Like a Same-Scale Phone Ensemble: The paper claims that the 6 streams capture different levels of linguistic abstraction (from fine phonetics to high-level semantics). However, the appendix data tells a simpler story:

  • Across layers 5–10 of SpidR 2.0, unit duration (70–80 ms), bitrate (90–100 bps), and phone purity (~60%) are almost flat (Figure 12).
  • Pairwise overlap between layers is very high (mean NMI of 0.71 in Figure 18, up from 0.55 in original SpidR).
  • Counter-intuitively, Phoneme ABX actually peaks in a deeper layer (Layer 10) than Word MAP (Layer 7) (Table 5).
  • In Table 12, the 6 individual stream heads inside GSLM2 get very similar scores on their own. Averaging them helps word and syntax scores (sWuggy +3.5%, sBLIMP +2.0%), but does not beat Layer 9 alone on semantics (tSC: 78.9% vs. 79.3%).

Together, this suggests GSLM2 is not learning a true coarse-to-fine hierarchy (like phones → syllables → words). Instead, it works like a multi-view ensemble at the same ~70 ms phone scale, where slightly different unit boundaries across layers regularize the shared backbone.

We agree that the SpidR 2.0 streams operate at a common temporal scale (unit durations of roughly 65–80 ms across layers 5–10). However, a shared rate does not imply shared content. Single-stream SLMs trained on each layer's units (see Figure 3) show opposite depth trends across tasks. From layer 5 to layer 10, sWuggy drops from 72.2 to 69.0 and sBLIMP from 63.8 to 61.7. Over the same layers, tSC rises by 1.5 points (and increases even at layer 12). Furthermore, layers 5–6 lag deeper layers on ABX (4.7 vs. 2.9% at layer 10) while being close to the best word MAP (67–68 vs. 69% at layer 7), indicating that streams at the same rate encode different information.

In addition, the stronger NMI between layers of SpidR 2.0 compared to SpidR can be explained by the fact that the deepest layers of SpidR collapse towards poor semantic representations, while layers 5 and 6 have an NMI of 0.71, which is very comparable to SpidR 2.0. Unrelated noisy layers have low NMI by construction, so low values observed in weak semantic representations are not a viable proxy to estimate complementarity in this case.

In Table 12, while averaging log-likelihoods across stream-specific heads does not beat the best one (layer 9), each head receives as input a shared context containing information from all streams. Also, when training a 300M / 6Kh single-stream SLM on layer 9 exclusively, we reach 77.8 on tSC, which is 1.5 points below layer 9’s stream-specific head in the multi-stream SLM at the same scale.

If GSLM2's gains came from slightly different unit boundaries regularizing a shared backbone, then any six-stream input with a matched amount of boundary disagreement should recover them. Appendix Table 13 tests this directly on the 6Kh / 300M setup. Copying the layer-7 argmax sequence into six streams matches the single-stream layer-7 SLM. Independently sampling six layer-7 streams from the codebook posteriors, with the temperature tuned to GSLM2's cross-stream compression ratio of 1.36, lowers the multi-stream SLM’s accuracy largely above SLM variance numbers.

However, we do agree with the reviewer that SpidR 2.0 does not learn a coarse-to-fine hierarchy through its layers, with phone-, syllable-, and word-sized units. To make this clearer and avoid confusion, we revised the relevant sentences in the abstract, introduction, and conclusion.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W2

~3x Compute Cost for a Mostly Constant Scaling Shift: Because cross-stream deduplication keeps a frame whenever any of the 6 streams changes, sequences are about 2x longer than in single-stream SpidR 2.0. Combined with the extra single-stream pass needed for self-distillation, GSLM2 uses 3x to 3.8x more training FLOPs and 2.4x to 3.4x more inference FLOPs per second (Table 10). Meanwhile, the scaling curves (Figure 6) show that upgrading the tokenizer to SpidR 2.0 drives most of the scaling slope improvement (Δγ=+0.016), while 6-stream modeling mostly gives a constant vertical shift (Δγ=+0.005).

We agree with the reviewer that GSLM2 is more expensive to train at the released configurations (3–3.8× more training FLOPs, Table 10), and that most of the slope improvement in SLM metrics comes from the SpidR 2.0 tokenizer. The newly added Appendix C.5 now analyzes this cost and compares both models at equal compute.

Where the cost comes from. For one training step at the 90M scale, GSLM2 requires 1.56×10151.56 \times 10^{15} FLOPs, against 4.09×10144.09 \times 10^{14} for SpidR 2.0 SLM. Of these:

  • the single-stream pass accounts for 23%, i.e., nearly a full SpidR 2.0 SLM step;
  • the multi-stream pass accounts for 62%, as cross-stream deduplication keeps a frame whenever any stream changes, which yields longer sequences, as the reviewer notes;
  • the generative head accounts for the remaining 15%, a share that decreases with model size.

These costs are hard to remove. The multi-stream pass is needed for inference and for the self-distillation loss, which is key to performance (Table 11). Its extra length comes precisely from the frames where faster streams change, i.e., the information that multi-stream modeling captures.

Why a constant shift matters. Speech LMs are also constrained by the amount of high-quality training data. In this regime, a constant improvement in accuracy corresponds to a multiplicative reduction in the data needed to reach a given performance. On the other hand, spending more compute on a single-stream SLM at fixed data does not buy this shift: increasing the batch size of SpidR 2.0 SLM to match the training FLOPs of GSLM2 does not improve its performance (setup A5 in Table 11).

We also run a comparison at equal training compute between multi- and single-stream SLMs, which illustrates the following:

  1. At the same training budget, GSLM2 trained on 6Kh comes within 0.6 points of the best single-stream SLM trained on 60Kh.
  2. The best GSLM2 configuration outperforms the best single-stream configuration on the average SLM score and matches it on LT-Swap, even though it had not yet converged.

In addition, our study in Appendix C.5 (see the new Figure 21) shows that above 102010^{20} training FLOPs, GSLM2 outperforms SpidR 2.0 SLM at equal compute.

We also encourage the reviewer to read our response to Q1 of Reviewer ym8U, which contains new results that we believe further strengthen our work.

Thank you for the response and the additional analyses in Appendix C.5. My concerns are addressed, and since my rating already points toward acceptance, I will maintain my current score.

Reviewer_UUhMOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents GSLM2, along with its encoder, SpidR 2.0. GSLM2 learns and models complementary properties of speech at multiple levels of abstraction, while SpidR 2.0 uses self-supervised learning with differentiable prediction heads to induce high-quality discrete units at multiple levels. A Multi-stream SLM would then model them jointly as time-aligned tuples. Across training data sizes and model scales, GSLM2 consistently improves linguistic competence while scaling more efficiently with data, which significantly narrows the gap between spoken and text-based LMs.

Critical strengths
  • The paper is clear and thorough.
  • The results are exceptionally strong.
Critical weaknesses

None.

Questions for authors
  • At 60Kh in Table 2, GSLM2 slightly outperforms Text LM on sBLI but falls behind by more than 15% on both IS and AS. Although the appendices (C.1.9 and C.1.12) do discuss the differences in grammatical coverage and possible TTS artifacts, can the authors further analyze what drives this discrepancy, for example, by comparing performance on the corresponding grammatical categories and excluding examples affected by synthesis artifacts?
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Response to Reviewer UUhM

We thank the reviewer for their positive assessment of our work and constructive feedback. We address each point below. The corresponding revisions, including additional analyses, are highlighted in blue in the revised manuscript.

Q1

At 60Kh in Table 2, GSLM2 slightly outperforms Text LM on sBLI but falls behind by more than 15% on both IS and AS. Although the appendices (C.1.9 and C.1.12) do discuss the differences in grammatical coverage and possible TTS artifacts, can the authors further analyze what drives this discrepancy, for example, by comparing performance on the corresponding grammatical categories and excluding examples affected by synthesis artifacts?

Thank you for your careful reading and this suggestion. We have conducted two complementary analyses to examine synthesis sensitivity and performance on the grammatical categories most relevant to Agreement/InflectionSwap, which are added in a new appendix section (Appendix C.1.13 "Speech Synthesis and Grammatical-Category Analysis for sBLIMP")

First, we resynthesized sBLIMP with Piper TTS, using the same synthesis configuration as spoken LT-Swap (see Appendix B.4.3). We also inspected some resynthesized samples and did not observe the specific artifacts identified in the original sBLIMP. As shown in Table A, at 60Kh, GSLM2’s overall accuracy decreases from 71.8% to 66.4%, and SpidR 2.0 SLM’s from 69.6% to 62.8%. Against the Text LM result of 71.5%, GSLM2 therefore trails by 5.1 percentage points, rather than matching it under the original synthesis. Both SLMs also obtain lower scores at 600h and 6Kh, but, importantly, their relative ranking remains consistent.

Table A. sBLIMP accuracy (%) on the original and Piper-resynthesized stimuli. Change denotes Piper minus original accuracy, in percentage points (pp).

Data / ParamsModelsBLIMP (org)sBLIMP (Piper)Change (pp)
600h / 90MSpidR 2.0 SLM57.755.0-2.7
GSLM259.856.7-3.1
Text LM61.7––
6Kh / 300MSpidR 2.0 SLM65.159.8-5.3
GSLM267.262.7-4.5
Text LM66.8––
60Kh / 1.2BSpidR 2.0 SLM69.662.8-6.8
GSLM271.866.4-5.4
Text LM71.5––

We then compared grammatically related subsets of Piper-resynthesized sBLIMP with AgreementSwap and InflectionSwap (Table B). For AgreementSwap, we selected subject-verb, determiner-noun, and reflexive-number agreement subtasks; for InflectionSwap, we considered the irregular past-participle subtasks involving adjectival and verbal uses. In Table C, at 60Kh, the gap between Text LM and GSLM2 is 13.0 points on sBLIMP (agreement) vs. 15.5 on AgreementSwap, and 14.9 points on sBLIMP (irregular forms) vs. 15.7 on InflectionSwap. Thus, comparing these related grammatical phenomena yields much closer text-speech gaps across benchmarks than comparing LT-Swap with the overall sBLIMP score.

Table B. Accuracy (%) on related grammatical categories in Piper-resynthesized sBLIMP and LT-Swap. LT-Swap scores are macro-averaged over word-frequency bins.

Data / ParamsModelAgreementSwapsBLIMP (agreement)InflectionSwapsBLIMP (irregular forms)
600h / 90MSpidR 2.0 SLM55.056.663.563.0
GSLM259.160.368.263.5
Text LM62.269.780.670.5
6Kh / 300MSpidR 2.0 SLM64.962.169.166.6
GSLM269.867.075.568.0
Text LM79.179.890.985.2
60Kh / 1.2BSpidR 2.0 SLM66.665.372.569.3
GSLM269.970.478.570.8
Text LM85.483.494.285.7

Table C. Accuracy (%) and Text LM–GSLM2 gaps on sBLIMP and LT-Swap at 60Kh. Gaps are computed as Text LM accuracy minus GSLM2 accuracy, in percentage points (pp).

Evaluation @ 60Kh / 1.2BGSLM2Text LMGap (pp)
sBLIMP (org; overall)71.871.5-0.3
sBLIMP (Piper; overall)66.471.55.1
sBLIMP (Piper; agreement)70.483.413.0
AgreementSwap69.985.415.5
sBLIMP (Piper; irregular forms)70.885.714.9
InflectionSwap78.594.215.7

Together, these analyses show that the apparent near-parity on the original sBLIMP is synthesis-sensitive and does not extend to the evaluated agreement and inflectional contrasts. We retain the original benchmark scores in the main tables for comparability with prior work, while explicitly qualifying their interpretation in the revised manuscript and reporting the additional resynthesis and category-level analyses in Appendix C.1.13.

Thank you for your thorough rebuttal. My concerns are resolved, and I will retain my score.

Reviewer_ym8UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new architecture for learning spoken language models (SLMs) directly from speech. The proposed model, named GSLM2, uses multiple streams of speech tokens, abstracted from intermediate layer representations from the speech encoder SpidR 2.0. A multi-stream SLM jointly learns the temporal and depth predictions of the unit streams. The paper presents the design of the framework, experiments across different datasets, and comparisons with existing SLMs trained from speech only. The proposed model shows strong performance and a better scaling trend.

Critical strengths

The paper is well written. The proposed framework design, learning of the model, and experiments and results are all well documented. Overall, its presentation is good.

Multi-stream design from intermediate encoder layers is also well motivated by previous works showing that different layer representations encode different features of the speech. The training laws also capture the dynamics between the temporal dimension as well as the depth dimension across different unit streams.

Experiments show promising results, with good comparisons with previously trained speech-only models.

Critical weaknesses

Proposed design of the LM induces more complexity and cost in training. The paper can benefit from more discussion on the cost and compute/flops controlled studies on the pretraining for comparison.

There are some choices between the multi-stream heads and generative heads on different evaluation protocols, and their application to real tasks needs more clarification. For example, Table 12 (in Appendix) shows that the generative head performs worse than the average of multi-stream heads for scoring, although the generative heads are used for generation. This may create some inconsistency of how the models should be used.

Questions for authors

Could the author provide comparisons with controlled flop during training? For example, how would their approach compare with a single-stream model with the same compute for pretraining?

For the scaling, the compute time and data are increased together. How are their contributions disentangled in the scaling behaviors?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Response to Reviewer ym8U

We thank the reviewer for their careful evaluation and for raising important questions, which we address each point below. The corresponding revisions, including additional analyses, are highlighted in blue in the revised manuscript.

W1

Proposed design of the LM induces more complexity and cost in training. The paper can benefit from more discussion on the cost and compute/flops controlled studies on the pretraining for comparison.

Thank you for your suggestion. Please see our response to Q1 for additional analyses, as well as the newly added Appendix C.5 in the revised manuscript.

W2

There are some choices between the multi-stream heads and generative heads on different evaluation protocols, and their application to real tasks needs more clarification. For example, Table 12 (in Appendix) shows that the generative head performs worse than the average of multi-stream heads for scoring, although the generative heads are used for generation. This may create some inconsistency of how the models should be used.

Thank you for pointing this out. GSLM2 is primarily intended for speech generation, for which we consistently use the generative head and assess generated continuations separately (Table 4). The paired-stimulus tasks serve as diagnostic probes of linguistic knowledge within the same jointly trained model.

For these probes, stream-specific heads predict each stream from the full multi-stream history, providing stream-wise readouts of the shared temporal backbone. We score unit-change positions and use the same unweighted aggregation across benchmarks. In contrast, the generative score includes persistence events and, for later streams, conditions on observed earlier-stream units within the current tuple. It therefore evaluates a different predictive quantity, including within-tuple dependencies.

Table 12 supports the transition-focused readout for linguistic probing, although it does not isolate the causes of the scoring gap. These diagnostic scores should not be equated with the generative head’s joint likelihood. Importantly, auxiliary stream-specific supervision also improves generation (Figure 9), supporting the complementary roles of the heads. In the revised manuscript, we have clarified their usage in Section 2.2.

Q1

Could the author provide comparisons with controlled flop during training? For example, how would their approach compare with a single-stream model with the same compute for pretraining?

We thank the reviewer for this suggestion. We added a new appendix section (Appendix C.5, "Training-Compute Efficiency Analysis") that investigates the decomposition of GSLM2’s training FLOPs.

At fixed data, increasing the training budget of single-stream SLMs to match the multi-stream often results in degraded performance, such as setup A5 in Table 11. Facing this unfair comparison, we decide to compare multi- and single-stream SLMs at controlled training compute in two ways, described in Appendix C.5.

First, we replot the results of the paper against total training FLOPs (Table 10). Below 102010^{20} FLOPs, it appears that single-stream SLMs are more compute-efficient: for instance, SpidR 2.0 SLM with 300M parameters trained on 6Kh outperforms GSLM2 with 90M parameters trained on 600h at a comparable budget. Above 102010^{20} FLOPs, GSLM2 outperforms SpidR 2.0 SLM at equal training compute.

However, since these configurations were not selected to be compute-optimal, we ran a new IsoFLOP experiment. We train SpidR 2.0 SLMs and GSLM2 models on the 6Kh and 60Kh Libri-Light subsets, all with the same training budget of 4×10194 \times 10^{19} FLOPs, the cost of GSLM2 with 90M parameters on 6Kh. The number of training steps is set by the budget, so the single-stream model can spend the same compute on a larger model or on more training steps. We find that:

  • Best configurations: GSLM2 outperforms the best single-stream configuration on the average SLM score (75.3 vs. 74.5) and matches it on LT-Swap, even though the best GSLM2 run had not yet converged at this budget.
  • Same data and model size: for model sizes around the optima, GSLM2 outperforms SpidR 2.0 SLM at the two studied data scales.
  • Data efficiency: GSLM2 trained on 6Kh comes within 0.6 points of the best single-stream model, which uses ten times more data.

This study is limited to one budget and a few model sizes, and a full compute-optimal analysis with several budgets is left for future work.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q2

For the scaling, the compute time and data are increased together. How are their contributions disentangled in the scaling behaviors?

Thank you for this question. Our main experiments indeed examine joint scaling of data, model capacity, and the optimization budget, which is motivated by the resource-scaling perspective of Cuervo and Marxer (2024). Since “compute time” may refer to different aspects of training resources, we address three related concerns below.

Training duration. If the concern is training longer, we increase optimization steps with dataset size (Table 9) to allow the models to learn from the additional examples. At a fixed batch size, maintaining comparable dataset coverage requires more updates. However, longer training does not necessarily improve validation performance: we select the checkpoint with the lowest validation loss, rather than automatically using the final checkpoint. For example, in the 600h setting, the best validation checkpoint is reached well before the scheduled end of training.

Model capacity. To assess the role of model capacity, we additionally evaluate fixed-90M backbones on the 600h, 6Kh, and 60Kh Libri-Light subsets (Table A). Both models improve as data grows, and GSLM2 outperforms SpidR 2.0 SLM on all three benchmarks at every scale. The 90M and 300M results are comparable at 6Kh. At 60Kh, scaling to 1.2B improves average accuracy over 90M by 2.4 percentage points for SpidR 2.0 SLM and 3.4 for GSLM2. Thus, gains persist without increasing backbone capacity, while larger-model configurations provide additional benefits at the largest data scale.

Table A. Fixed-90M and scaled-up results of SpidR 2.0 SLM and GSLM2.

Model#Data [h]#ParamssWuggysBLIMPtSCAvg.
SpidR 2.0 SLM60090M69.257.768.165.1
6K90M73.765.475.371.5
6K300M73.965.174.771.2
60K90M75.467.777.173.4
60K1.2B76.769.681.075.8
GSLM260090M71.559.874.168.5
6K90M76.167.478.474.0
6K300M76.367.278.974.1
60K90M77.268.579.675.1
60K1.2B79.371.884.578.5

Total compute budget. If the concern is performance at a fixed overall training-compute budget, please see the additional matched-training-FLOPs comparisons in our response to Q1.

In the revised manuscript, in Section 3.2 and Appendix C.1.2, we have clarified that our scaling studies characterize performance under the observed joint scaling schedule, rather than isolating the effect of data at fixed capacity and compute.

Dear Reviewer ym8U,

As the discussion period closes on November 18, we would welcome your feedback on our response. We would be happy to clarify any points or address any remaining questions or concerns.

Thank you for your time and consideration.

I thank the authors for addressing my concerns and questions. The scaling experiments across dataset size and model sizes are very informative. I will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.