acceptodds
Under review as a conference paper at ICLR 2027

Matryoshka Language Model Suites

20 views

Abstract

Training a language model suite classically requires training each model separately and serving them independently. We improve both training and inference efficiency by stacking sub-models of increasing size into a single nested architecture trained end-to-end. This Matryoshka training framework reduces the total parameter count of the suite, enables low-cost distillation from the largest to all smaller sub-models at every training step, and is well-suited for speculative decoding as the draft model is contained within the verifier. We validate our approach by training a Matryoshka suite comprising 500M, 1.5B, and 3B sub-models. Our suite is on par with independently trained baselines on validation perplexity, out-of-domain perplexity, and benchmark performance while using 36% less training compute, and achieves 14-26% higher throughput via speculative decoding. We also ablate key architectural choices, offering guidance for building strong Matryoshka LM suites.

Primary area:
foundation or frontier models, including LLMs
Keywords:
pretraining, model suites, modularity, efficiency, architecture, speculative decoding

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_6qi5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Matryoshka LM Suites nests sub-models of increasing size (500M / 1.5B / 3B) into a single jointly pretrained architecture: each sub-model has its own width, depth and LM head, and a junction mechanism (norm-rescaled output concatenated with a fresh embedding) passes representations between stages. Distillation from the largest sub-model to all smaller ones is obtained for free at every forward pass, and detachable sub-models share KV cache with the verifier, making the design a natural fit for speculative decoding. On 35B FineWeb-Edu tokens, the 3B suite is reported to reach near-parity with independently trained baselines using 36% less training compute, and to improve speculative-decoding throughput by 14-26%.

Critical strengths

Well-motivated problem (joint suite training is genuinely understudied). Careful architecture design: depth triplets are selected by matching both KV cache and per-token FLOPs to the Vanilla baseline rather than by quality metrics. Extensive proxy-scale ablations covering the junction components, distillation, and width/depth allocation. Broad evaluation surface: 7 benchmarks, 5 OOD PPL corpora, Tulu-3 SFT, speculative decoding, and a MatFormer comparison.

Critical weaknesses
  1. The advantage over the Token-matched Vanilla baseline is modest and warrants stronger evidence. In Table 2, Matryoshka's average accuracy is slightly below Token-matched Vanilla at every size (500M 46.5 vs 47.0; 1.5B 52.2 vs 52.4; 3B 53.3 vs 53.5). The 36% compute saving is a meaningful practical benefit, but the quality tradeoff should be characterized with multi-seed runs at the 3B scale; currently the headline is single-seed and noise is only estimated at the 200M proxy (~0.1% PPL).

  2. Positioning against MatFormer / LayerSkip / Flextron would be stronger at the main scale. The MatFormer comparison runs only at the 200M proxy, not at 3B where the headline claims live. A head-to-head with LayerSkip or Flextron/LLaMaFlex at matched budgets would sharpen the contribution. The junction mechanism and online distillation are natural compositions of existing components; the novelty is more in the system design than in individual primitives.

  3. Speculative-decoding gains would be more convincing under symmetric conditions. The 26% speedup uses batch=102 for Matryoshka vs batch=64 for Vanilla; at matched batch size (Appendix L) the gain is 9-18%, still meaningful but less dramatic. The Vanilla baseline (500M/3B) is acknowledged to "barely beat autoregressive"; a comparison against a smaller draft or Vanilla+EAGLE would better represent the practical alternative.

Questions for authors
  1. Can you provide multi-seed 3B results with error bars? Given that Matryoshka is 0.2-0.5 accuracy points below Token-matched Vanilla at every size, the single-seed headline leaves the "near-parity" claim underpowered.

  2. Head-to-head with MatFormer at the 3B scale (not only the 200M proxy)?

  3. Speculative-decoding results at matched batch size as the main number, plus a Matryoshka+EAGLE vs Vanilla+EAGLE comparison?

3 comments

We thank the reviewer for their detailed and constructive feedback.

Can you provide multi-seed 3B results with error bars? Given that Matryoshka is 0.2-0.5 accuracy points below Token-matched Vanilla at every size, the single-seed headline leaves the "near-parity" claim underpowered

Each pretraining comparison run costs roughly 1,000 B200 GPU-hours. We cannot run such experiments with our hardware within the time limits of the discussion period. Given that pretraining is known to be very consistent across seeds [1], which we verify at 200M scale (5 seeds per suite, Appendix M), we do not believe that multiple seeds would noticeably affect the downstream results.

We re-ran our NLU evaluations to get sample-level results, and we use statistical tests to measure the significance of two claims: Matryoshka models are significantly better than the compute-matched Vanilla models (=FLOPs), and token-matched Vanilla models (=Tokens) are not significantly better than the Matryoshka models.

SizeMatryoshkaVanilla (=Tokens)Vanilla (=FLOPs)p (Matryoshka > =FLOPs)p (=Tokens > Matryoshka)
500M46.747.146.10.0740.12
1.5B52.252.450.4<1e-60.37
3B53.353.652.1<1e-30.21

The p-values come from one-sided paired tests on the per-example differences between the two models over the benchmark examples. Bold p-values support our claims: p<0.05 in the first column means Matryoshka is significantly better than Vanilla (=FLOPs); p>0.05 in the second means Vanilla (=Tokens) is not significantly better than Matryoshka.

The gaps between Matryoshka models and their compute-matched baselines are statistically significant, except at the 500M size, where the p-value is still close to 0.05. However, at no size is the token-matched Vanilla model significantly stronger than Matryoshka. Hence, these results support our claims statistically.

Head-to-head with MatFormer at the 3B scale (not only the 200M proxy)?

We are unable to run these 3B-scale pretraining experiments within the discussion period. In the original MatFormer paper [2], Figure 3a shows that the XL MatFormer model scales slightly worse than vanilla when going from 100M to 8B scale. Given that we find that MatFormer slightly lags behind both our vanilla baseline and the largest Matryoshka model at 200M scale, we hypothesize it is unlikely to outperform both at larger scale.

Speculative-decoding results at matched batch size as the main number, plus a Matryoshka+EAGLE vs Vanilla+EAGLE comparison?

The fact that Matryoshka LM suites allow larger batch sizes in our speculative setup is not a side effect of our method but rather an expected benefit of model nesting, which is why we keep the matched-memory setting as our primary benchmark. Our 14-26% throughput numbers correspond to maximal throughput speedups across decoding setups (sampling method and draft length), with each pair evaluated at its own maximum feasible batch size. As the reviewer remarks, the gain remains meaningful at matched batch size (1.09–1.18×, Appendix L).

We could not train an EAGLE head on top of the Vanilla model within the discussion period. However, we address the complementarity of Matryoshka LM suites with EAGLE in Appendix K, where we show that EAGLE provides benefits to the Matryoshka 3B model similar to those reported in the original paper (roughly doubling throughput), and that the smaller submodel can be used as a pre-verifier to accelerate inference at larger batch sizes. This shows that EAGLE is compatible with Matryoshka submodels, and that we can benefit from the nested structure to provide further speedup.

[1] PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs, van der Wal et al., 2025

[2] MatFormer: Nested Transformer for Elastic Inference, Devvrit et al., 2024

Reviewer_6qi5OpenReview ↗

Response to Authors

Thank you for the detailed response and clarifications. My main concerns have been largely addressed, and I appreciate the authors’ efforts to resolve them.

Thank you for the nice comments, and we are glad to have addressed your concerns! If any point still holds back updating the numerical assessment, we would be happy to clarify it within the remaining discussion period.

Reviewer_tSb2OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary
  1. In this work, the authors introduce a new framework for training models nested within each other similar to matryoshka. The authors claim that the proposed framework allows end to end training of the nested models and improves both the training and inference efficiency
  2. The framework allows distilling multiple nested models simultaneously from a large model as well as is suitable for speculative decoding as the small model is contained withing the larger verifier model
  3. The authors perform experiments for models upto 3B models to demonstrate the efficiency and speculative decoding throughput improvements.
Critical strengths

The primary goal of introducing a suite is to avoid separate training of multiple smaller models while training a larger model. The problem is well motivated, the writing is clear. The appoach primarily draws from the matryoshka representation learning and matformer works but has some really interesting contributions and experiments like:

  1. nesting with different widths and depths can be stacked and nested to form larger models.
  2. the ability to train smaller models below a threshold, flexibility with the KV cache.
  3. intermodel junction modeling
  4. ability to distill from the larger models from the suite into a smaller one and speculative decoding with a larger model as a verfier and a smaller model as a drafter.
Critical weaknesses

I did not find any major weaknesses

Questions for authors

[1] In figure 1(b) I think instead of validation perplexity, reporting the averaged benchmark performance would be very helpful. [2] (clarifying question) What do the two arrows in Figure 1(a) show?

1 comment

We are grateful to Reviewer tSb2 for their very positive assessment of our work, and we thank tSb2 for the valuable recommendations.

[1] Benchmark accuracy in Figure 1(b)

Thank you for this suggestion. We chose to report validation perplexity in Figure 1(b) because it directly reflects the pretraining objective and is less noisy than zero-shot benchmark accuracy at these model scales, where differences between models are often within evaluation noise. Average benchmark accuracy is reported for each sub-model size in Table 2 and throughout training in Figure 3, where Matryoshka sub-models stay within 0.5 points of the Vanilla models trained on the same number of tokens, and outperform the compute-matched Vanilla models by 0.4 to 1.9 points. We added a pointer to these results in the caption of Figure 1 in the revised version.

[2] (clarifying question) What do the two arrows in Figure 1(a) show?

These arrows illustrate the gaps between Vanilla suites and Matryoshka suites. If the reviewer deems it necessary, we could specify this in the legend.

Reviewer_xdC9OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a practical and effective approach to jointly training language models of different sizes. The reported 36% training compute reduction and 14–26% speculative decoding throughput improvement are compelling. The experimental evaluation is reasonably strong. However, larger-scale validation, compute-matched comparisons, and broader inference evaluations would further strengthen the work.

Critical strengths

This is a practically relevant problem with meaningful systems implications. The paper evaluates 500M, 1.5B, and 3B models trained on 35B FineWeb-Edu tokens. It reports a 36% reduction in total suite training compute, near-parity with independently trained models, and 14–26% improvement in speculative-decoding throughput. It also investigates architectural junctions, distillation, and width/depth choices.

Critical weaknesses
  • The comparison against independently trained suites is compelling, but the distinction between compute-matched and token-matched baselines is crucial. Token-matched vanilla models receive more compute.

  • Training at 3B parameters is useful evidence, but insufficient to establish efficiency at modern frontier-model scales or in MoE architectures.

  • The shared-weight design creates dependencies among submodels. The paper should quantify how much independent architectural and data optimization is sacrificed.

Questions for authors
  • How does the training frontier compare against independently trained, optimally sized models at a fixed total compute budget?
  • What is the benefit on realistic high-throughput serving workloads?
  • Does the architecture extend efficiently to MoE models?
1 comment

The shared-weight design creates dependencies among submodels. The paper should quantify how much independent architectural and data optimization is sacrificed.

These dependencies are inherent to nesting, but our results show that they come at little cost in quality: Matryoshka submodels reach a similar quality to Vanilla models trained on the same number of tokens, which do not have this dependency constraint.

From a design space viewpoint, it is true that we only explore a single architecture type and data mixture across submodels in our experiments. However, it is possible to design extensions of our work that do not share this monolithic design. From the architectural viewpoint, the only constraint we impose is a non-decreasing hidden dimension as submodels grow in size; in theory, each submodel layer stack could adopt completely different internal architecture choices. From the data viewpoint, concurrent training imposes a common batch distribution, but we hypothesize that weighting samples differently in each submodel's loss computation could approximate separate data mixtures. Thus, we argue that our approach does not imply strong conceptual sacrifices to either data or architecture optimization, and we will discuss these points in the revised version.

How does the training frontier compare against independently trained, optimally sized models at a fixed total compute budget?

Our setup follows the usual suite pretraining regime, where model sizes are fixed by deployment constraints and models are trained on a common data mixture. In this context, all sizes are trained on the same token budget rather than being sized for compute-optimality. At a fixed total compute budget, our compute-matched Vanilla baseline (=FLOPs) shows that Matryoshka outperforms independently trained models of the same sizes.

Identifying optimal sizes at a fixed total compute budget would require us to measure scaling laws for vanilla models in our setup, but also to define and estimate scaling laws for Matryoshka suites. This is an interesting problem, but it entails experiments that cannot be run within the discussion period, and that fall outside the scope of our paper.

What is the benefit on realistic high-throughput serving workloads?

Our speculative decoding experiments already cover high-throughput regimes, as we measure throughput across different batch sizes, up to memory saturation. In this regime, KV cache sharing lets Matryoshka reach a batch size of 160 while the Vanilla pair runs out of memory from 148 on a GH200 GPU, for a 1.32× end-to-end throughput gain (Appendix L).

Does the architecture extend efficiently to MoE models?

We do not see a fundamental obstacle to combining MoEs and Matryoshka LM suites. Our nesting operates on blocks of layers, and the FFN layers within each block could be replaced with MoE layers without changing the sub-model construction. We expect compounding efficiency benefits, as these methods are fairly orthogonal. We leave this exploration for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.