- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper aims to close the large performance and data-efficiency gap between textless Spoken Language Models (SLMs) and Text LMs. The authors point out two main bottlenecks in current SLMs:
- Discrete speech units are noisy, unstable across random seeds, and require a separate post-hoc -means step.
- Standard SLMs only use units from a single encoder layer, forcing a trade-off between fine phonetic detail and word-level meaning.
To fix these issues, the authors propose GSLM2, which has two main parts:
- SpidR 2.0 (Speech Tokenizer): A self-supervised student-teacher encoder that learns discrete units directly during training using shared, normalized prediction heads and entropy regularization (ME-MAX). This removes -means and creates stable, longer (~70–90 ms) phone-like units across multiple layers.
- Multi-Stream SLM: A language model that jointly processes 6 unit streams (from layers 5 to 10 of SpidR 2.0) after removing repeated frames across streams. It uses a shared Transformer backbone with two types of heads:
- Stream-specific linear heads that predict when a stream changes to a new unit (trained with a self-distillation loss from a single-stream pass).
- An autoregressive depth Transformer head that generates the 6-stream unit tuple at each step.
The authors test GSLM2 on Libri-Light (up to 60K hours / 1.2B parameters), a 6M-hour conversational speech dataset (up to 6.5B parameters), and multilingual VoxPopuli data (English, French, German). GSLM2 beats single-stream baselines and larger speech/speech-text models (such as Moshi and Spirit-LM), cutting the data-scaling gap to Text LMs from 4 orders of magnitude down to 2.
Critical strengths
-
Cleaner, More Stable Speech Units Without -Means (SpidR 2.0): Removing post-hoc -means and learning normalized codebooks directly during pretraining works very well. It cuts cross-seed ABX variance by , reaches a strong 2.83% triphone ABX error, and keeps high quality across layers 5 to 12 instead of peaking at just one layer (Figure 3). It also naturally compresses speech into longer (~70–90 ms) units at roughly half the bitrate, which makes language modeling much easier.
-
Effective Multi-Stream Modeling and Self-Distillation: Combining 6 layers consistently beats using any single layer. The authors include great control tests (Table 13) proving that simply repeating or sampling from the best layer (Layer 7) six times does not give the same boost. They also show through ablations that their self-distillation loss ()—which aligns multi-stream predictions with a single-stream pass—is the key ingredient, adding +7.8 to +9.9 accuracy points (Table 11).
-
Thorough Scaling Study and Useful New Benchmarks: The paper scales up to 6M hours of real conversational speech and 6.5B parameters, with fair Text LM and cascaded ASR+LM baselines. It also introduces valuable new benchmarks (Spoken LongTail-Swap and French/German evaluations), multi-seed variance breakdowns, and a 20-encoder study showing that Word MAP predicts downstream SLM performance much better than ABX or PNMI.
Critical weaknesses
-
"Different Levels of Abstraction" Looks More Like a Same-Scale Phone Ensemble: The paper claims that the 6 streams capture different levels of linguistic abstraction (from fine phonetics to high-level semantics). However, the appendix data tells a simpler story:
- Across layers 5–10 of SpidR 2.0, unit duration (70–80 ms), bitrate (90–100 bps), and phone purity (~60%) are almost flat (Figure 12).
- Pairwise overlap between layers is very high (mean NMI of 0.71 in Figure 18, up from 0.55 in original SpidR).
- Counter-intuitively, Phoneme ABX actually peaks in a deeper layer (Layer 10) than Word MAP (Layer 7) (Table 5).
- In Table 12, the 6 individual stream heads inside GSLM2 get very similar scores on their own. Averaging them helps word and syntax scores (sWuggy +3.5%, sBLIMP +2.0%), but does not beat Layer 9 alone on semantics (tSC: 78.9% vs. 79.3%).
- Together, this suggests GSLM2 is not learning a true coarse-to-fine hierarchy (like phones syllables words). Instead, it works like a multi-view ensemble at the same ~70 ms phone scale, where slightly different unit boundaries across layers regularize the shared backbone.
-
~ Compute Cost for a Mostly Constant Scaling Shift: Because cross-stream deduplication keeps a frame whenever any of the 6 streams changes, sequences are about longer than in single-stream SpidR 2.0. Combined with the extra single-stream pass needed for self-distillation, GSLM2 uses to more training FLOPs and to more inference FLOPs per second (Table 10). Meanwhile, the scaling curves (Figure 6) show that upgrading the tokenizer to SpidR 2.0 drives most of the scaling slope improvement (), while 6-stream modeling mostly gives a constant vertical shift ().
Questions for authors
see weaknesses
3 comments
Rebuttal by Authors (2/2)
W2
~3x Compute Cost for a Mostly Constant Scaling Shift: Because cross-stream deduplication keeps a frame whenever any of the 6 streams changes, sequences are about 2x longer than in single-stream SpidR 2.0. Combined with the extra single-stream pass needed for self-distillation, GSLM2 uses 3x to 3.8x more training FLOPs and 2.4x to 3.4x more inference FLOPs per second (Table 10). Meanwhile, the scaling curves (Figure 6) show that upgrading the tokenizer to SpidR 2.0 drives most of the scaling slope improvement (Δγ=+0.016), while 6-stream modeling mostly gives a constant vertical shift (Δγ=+0.005).
We agree with the reviewer that GSLM2 is more expensive to train at the released configurations (3–3.8× more training FLOPs, Table 10), and that most of the slope improvement in SLM metrics comes from the SpidR 2.0 tokenizer. The newly added Appendix C.5 now analyzes this cost and compares both models at equal compute.
Where the cost comes from. For one training step at the 90M scale, GSLM2 requires FLOPs, against for SpidR 2.0 SLM. Of these:
- the single-stream pass accounts for 23%, i.e., nearly a full SpidR 2.0 SLM step;
- the multi-stream pass accounts for 62%, as cross-stream deduplication keeps a frame whenever any stream changes, which yields longer sequences, as the reviewer notes;
- the generative head accounts for the remaining 15%, a share that decreases with model size.
These costs are hard to remove. The multi-stream pass is needed for inference and for the self-distillation loss, which is key to performance (Table 11). Its extra length comes precisely from the frames where faster streams change, i.e., the information that multi-stream modeling captures.
Why a constant shift matters. Speech LMs are also constrained by the amount of high-quality training data. In this regime, a constant improvement in accuracy corresponds to a multiplicative reduction in the data needed to reach a given performance. On the other hand, spending more compute on a single-stream SLM at fixed data does not buy this shift: increasing the batch size of SpidR 2.0 SLM to match the training FLOPs of GSLM2 does not improve its performance (setup A5 in Table 11).
We also run a comparison at equal training compute between multi- and single-stream SLMs, which illustrates the following:
- At the same training budget, GSLM2 trained on 6Kh comes within 0.6 points of the best single-stream SLM trained on 60Kh.
- The best GSLM2 configuration outperforms the best single-stream configuration on the average SLM score and matches it on LT-Swap, even though it had not yet converged.
In addition, our study in Appendix C.5 (see the new Figure 21) shows that above training FLOPs, GSLM2 outperforms SpidR 2.0 SLM at equal compute.
We also encourage the reviewer to read our response to Q1 of Reviewer ym8U, which contains new results that we believe further strengthen our work.
Thank you for the response and the additional analyses in Appendix C.5. My concerns are addressed, and since my rating already points toward acceptance, I will maintain my current score.
Rebuttal by Authors (1/2)
Response to Reviewer Pwrk
We appreciate the reviewer's time and effort in providing a comprehensive review and raising thoughtful questions, which we address point by point below. The corresponding revisions, including additional analyses, are highlighted in blue in the revised manuscript.
W1
We agree that the SpidR 2.0 streams operate at a common temporal scale (unit durations of roughly 65–80 ms across layers 5–10). However, a shared rate does not imply shared content. Single-stream SLMs trained on each layer's units (see Figure 3) show opposite depth trends across tasks. From layer 5 to layer 10, sWuggy drops from 72.2 to 69.0 and sBLIMP from 63.8 to 61.7. Over the same layers, tSC rises by 1.5 points (and increases even at layer 12). Furthermore, layers 5–6 lag deeper layers on ABX (4.7 vs. 2.9% at layer 10) while being close to the best word MAP (67–68 vs. 69% at layer 7), indicating that streams at the same rate encode different information.
In addition, the stronger NMI between layers of SpidR 2.0 compared to SpidR can be explained by the fact that the deepest layers of SpidR collapse towards poor semantic representations, while layers 5 and 6 have an NMI of 0.71, which is very comparable to SpidR 2.0. Unrelated noisy layers have low NMI by construction, so low values observed in weak semantic representations are not a viable proxy to estimate complementarity in this case.
In Table 12, while averaging log-likelihoods across stream-specific heads does not beat the best one (layer 9), each head receives as input a shared context containing information from all streams. Also, when training a 300M / 6Kh single-stream SLM on layer 9 exclusively, we reach 77.8 on tSC, which is 1.5 points below layer 9’s stream-specific head in the multi-stream SLM at the same scale.
If GSLM2's gains came from slightly different unit boundaries regularizing a shared backbone, then any six-stream input with a matched amount of boundary disagreement should recover them. Appendix Table 13 tests this directly on the 6Kh / 300M setup. Copying the layer-7 argmax sequence into six streams matches the single-stream layer-7 SLM. Independently sampling six layer-7 streams from the codebook posteriors, with the temperature tuned to GSLM2's cross-stream compression ratio of 1.36, lowers the multi-stream SLM’s accuracy largely above SLM variance numbers.
However, we do agree with the reviewer that SpidR 2.0 does not learn a coarse-to-fine hierarchy through its layers, with phone-, syllable-, and word-sized units. To make this clearer and avoid confusion, we revised the relevant sentences in the abstract, introduction, and conclusion.