- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Matryoshka LM Suites nests sub-models of increasing size (500M / 1.5B / 3B) into a single jointly pretrained architecture: each sub-model has its own width, depth and LM head, and a junction mechanism (norm-rescaled output concatenated with a fresh embedding) passes representations between stages. Distillation from the largest sub-model to all smaller ones is obtained for free at every forward pass, and detachable sub-models share KV cache with the verifier, making the design a natural fit for speculative decoding. On 35B FineWeb-Edu tokens, the 3B suite is reported to reach near-parity with independently trained baselines using 36% less training compute, and to improve speculative-decoding throughput by 14-26%.
Critical strengths
Well-motivated problem (joint suite training is genuinely understudied). Careful architecture design: depth triplets are selected by matching both KV cache and per-token FLOPs to the Vanilla baseline rather than by quality metrics. Extensive proxy-scale ablations covering the junction components, distillation, and width/depth allocation. Broad evaluation surface: 7 benchmarks, 5 OOD PPL corpora, Tulu-3 SFT, speculative decoding, and a MatFormer comparison.
Critical weaknesses
-
The advantage over the Token-matched Vanilla baseline is modest and warrants stronger evidence. In Table 2, Matryoshka's average accuracy is slightly below Token-matched Vanilla at every size (500M 46.5 vs 47.0; 1.5B 52.2 vs 52.4; 3B 53.3 vs 53.5). The 36% compute saving is a meaningful practical benefit, but the quality tradeoff should be characterized with multi-seed runs at the 3B scale; currently the headline is single-seed and noise is only estimated at the 200M proxy (~0.1% PPL).
-
Positioning against MatFormer / LayerSkip / Flextron would be stronger at the main scale. The MatFormer comparison runs only at the 200M proxy, not at 3B where the headline claims live. A head-to-head with LayerSkip or Flextron/LLaMaFlex at matched budgets would sharpen the contribution. The junction mechanism and online distillation are natural compositions of existing components; the novelty is more in the system design than in individual primitives.
-
Speculative-decoding gains would be more convincing under symmetric conditions. The 26% speedup uses batch=102 for Matryoshka vs batch=64 for Vanilla; at matched batch size (Appendix L) the gain is 9-18%, still meaningful but less dramatic. The Vanilla baseline (500M/3B) is acknowledged to "barely beat autoregressive"; a comparison against a smaller draft or Vanilla+EAGLE would better represent the practical alternative.
Questions for authors
-
Can you provide multi-seed 3B results with error bars? Given that Matryoshka is 0.2-0.5 accuracy points below Token-matched Vanilla at every size, the single-seed headline leaves the "near-parity" claim underpowered.
-
Head-to-head with MatFormer at the 3B scale (not only the 200M proxy)?
-
Speculative-decoding results at matched batch size as the main number, plus a Matryoshka+EAGLE vs Vanilla+EAGLE comparison?
3 comments
Response to Authors
Thank you for the detailed response and clarifications. My main concerns have been largely addressed, and I appreciate the authors’ efforts to resolve them.
Thank you for the nice comments, and we are glad to have addressed your concerns! If any point still holds back updating the numerical assessment, we would be happy to clarify it within the remaining discussion period.
We thank the reviewer for their detailed and constructive feedback.
Can you provide multi-seed 3B results with error bars? Given that Matryoshka is 0.2-0.5 accuracy points below Token-matched Vanilla at every size, the single-seed headline leaves the "near-parity" claim underpowered
Each pretraining comparison run costs roughly 1,000 B200 GPU-hours. We cannot run such experiments with our hardware within the time limits of the discussion period. Given that pretraining is known to be very consistent across seeds [1], which we verify at 200M scale (5 seeds per suite, Appendix M), we do not believe that multiple seeds would noticeably affect the downstream results.
We re-ran our NLU evaluations to get sample-level results, and we use statistical tests to measure the significance of two claims: Matryoshka models are significantly better than the compute-matched Vanilla models (=FLOPs), and token-matched Vanilla models (=Tokens) are not significantly better than the Matryoshka models.
The p-values come from one-sided paired tests on the per-example differences between the two models over the benchmark examples. Bold p-values support our claims: p<0.05 in the first column means Matryoshka is significantly better than Vanilla (=FLOPs); p>0.05 in the second means Vanilla (=Tokens) is not significantly better than Matryoshka.
The gaps between Matryoshka models and their compute-matched baselines are statistically significant, except at the 500M size, where the p-value is still close to 0.05. However, at no size is the token-matched Vanilla model significantly stronger than Matryoshka. Hence, these results support our claims statistically.
Head-to-head with MatFormer at the 3B scale (not only the 200M proxy)?
We are unable to run these 3B-scale pretraining experiments within the discussion period. In the original MatFormer paper [2], Figure 3a shows that the XL MatFormer model scales slightly worse than vanilla when going from 100M to 8B scale. Given that we find that MatFormer slightly lags behind both our vanilla baseline and the largest Matryoshka model at 200M scale, we hypothesize it is unlikely to outperform both at larger scale.
Speculative-decoding results at matched batch size as the main number, plus a Matryoshka+EAGLE vs Vanilla+EAGLE comparison?
The fact that Matryoshka LM suites allow larger batch sizes in our speculative setup is not a side effect of our method but rather an expected benefit of model nesting, which is why we keep the matched-memory setting as our primary benchmark. Our 14-26% throughput numbers correspond to maximal throughput speedups across decoding setups (sampling method and draft length), with each pair evaluated at its own maximum feasible batch size. As the reviewer remarks, the gain remains meaningful at matched batch size (1.09–1.18×, Appendix L).
We could not train an EAGLE head on top of the Vanilla model within the discussion period. However, we address the complementarity of Matryoshka LM suites with EAGLE in Appendix K, where we show that EAGLE provides benefits to the Matryoshka 3B model similar to those reported in the original paper (roughly doubling throughput), and that the smaller submodel can be used as a pre-verifier to accelerate inference at larger batch sizes. This shows that EAGLE is compatible with Matryoshka submodels, and that we can benefit from the nested structure to provide further speedup.
[1] PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs, van der Wal et al., 2025
[2] MatFormer: Nested Transformer for Elastic Inference, Devvrit et al., 2024