acceptodds
Under review as a conference paper at ICLR 2027

ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer

Abstract

Foundation models need to support diverse inference budgets, yet training a separate model for each budget duplicates pretraining effort. Elastic models train several nested sub-networks inside one weight-shared supernet, but existing recipes face three obstacles: variable per-step workloads that hinder distributed scaling, limited hyperparameter (HP) transfer across model sizes, and no guarantee that all sub-networks are maximally updated at once. We present ShellElastic, an elastic transformer that organizes hidden dimensions into a dense core and successive shells, so that every hidden matrix is block lower triangular. One full-width forward pass contains every sub-network's exact hidden states as prefixes, so joint training keeps a fixed computation graph and costs about as much per step as training the widest sub-network alone. We further introduce -Elastic, an extension of maximal update parameterization (P) to ShellElastic and other elastic architectures. Its scaling rules provably keep all sub-networks of a supernet maximally updated simultaneously (-Internal), and keep doing so when the supernet is widened or given more shells; sweeps confirm that HPs tuned on a small non-elastic proxy stay near-optimal for elastic families and larger supernets (-Transfer). A single ShellElastic run produces sub-networks that outperform MatFormer and fully elastic baselines at matched parameter counts, and approach separately trained models that together take several times the steps.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Pretraining, Foundation Models, Elastic Models, Maximal Update Parametrization, Hyperparameter Transfer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ag77OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ShellElastic, an elastic Transformer pretraining framework designed to support different inference budgets. ShellElastic organizes hidden dimensions into a dense core and successive shells, using block lower triangular projections to enable exact hidden-state reuse across all supported sub-networks within a single full-width forward pass. The authors further introduce μ-Elastic, an extension of maximal update parameterization (μP) to elastic models, enabling simultaneous maximal updates and hyperparameter transfer. Experiments across three model scales show that ShellElastic outperforms MatFormer and fully elastic baselines in validation loss and downstream performance, while approaching the performance of separately trained models.

Critical strengths
  1. ShellElastic achieves exact computation reuse across sub-networks through block lower triangular projections and group-local normalization. μ-Elastic further provides hyperparameter scaling rules for shared parameters, supported by theoretical analysis of simultaneous maximal updates and hyperparameter transfer.

  2. The paper evaluates the method across three model scales and compares it with several elastic training baselines. The hyperparameter transfer experiments, architecture search, and comprehensive ablation studies provide substantial empirical support for the proposed designs.

Critical weaknesses
  1. ShellElastic achieves strong performance through a relatively large dense core and a specific configuration of supported widths. However, this design may limit its flexibility in accommodating different inference budgets. Although the paper includes architecture search experiments, the trade-off between model performance and the range of supported inference budgets remains insufficiently discussed, which may affect its practical applicability.

  2. Although the paper demonstrates promising results across multiple model scales, the experiments are still limited to relatively small model sizes and a restricted set of architectural configurations. Whether ShellElastic and μ-Elastic can consistently maintain their performance, training efficiency, and hyperparameter transfer advantages at larger scales and with more supported widths remains to be established.

Questions for authors
  1. Can the authors further discuss how the choice of dense core size and supported widths affects the trade-off between model performance and elasticity, and how this trade-off can be balanced under different inference budgets?

  2. How are the performance and training efficiency of ShellElastic and μ-Elastic expected to scale with larger model sizes and an increasing number of supported widths?

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer ag77 (Part 1)

We thank Reviewer ag77 for the insightful comments and suggestions, and for the overall positive assessment of our paper. We appreciate that you find that ShellElastic achieves exact computation reuse across sub-networks, that μ\mu-Elastic provides hyperparameter scaling rules for shared parameters with theoretical analysis of simultaneous maximal updates and hyperparameter transfer, and that our evaluation, hyperparameter transfer experiments, architecture search and comprehensive ablation studies provide substantial empirical support for our designs. We address your concerns point by point below.

W1&Q1 Trade-off between performance and supported widths

The core size and the supported widths indeed trade off performance against elasticity, and they are not fixed to the configuration in the paper (a 512-wide core followed by four 128-wide shells). In practice, we first set the total parameters and the number of supported widths according to the budgets required for deployment. The practical design principle is then to put more parameters into the core, as this gives the best overall performance. Our architecture search keeps the total parameters and the number of widths fixed and only changes how the hidden dimensions are split between the core and the shells (Section 4.3 and Appendix J.2). A large core with thin shells has the lowest validation loss, or is within 0.006 nats of it, at every width, whereas a wide shell makes the smallest width that includes it 0.025-0.052 nats worse.

Every added width costs the other widths some performance, and this cost is much smaller for ShellElastic than for sample-based methods such as MatFormer and Fully Elastic. This is because the loss of each sub-network follows how often it is trained and its loss weight (Appendix K(1) and K(2)). A sample-based method trains one sampled width per step, so every added width makes each width trained less often. ShellElastic instead trains all widths at every step, so an added width only reduces the loss weight of the others. With a shared readout, an added width also adds no FLOPs per step (see W2&Q2).

We train ShellElastic with nine widths from 512 to 1024 in steps of 64 and compare it with five widths over the same range in steps of 128, both with a base width of 64 (Appendix K(11)) and the hyperparameters transferred from the proxy. The results are shown in Table T1. The four added widths lie between their neighbours in validation loss. The five shared widths keep their 5-shot accuracy (36.1 vs. 35.7 on average), and their validation loss rises by only 0.007 nats on average, mostly at width 512. This is because each smaller width gets 1/16 of the loss weight with nine widths instead of 1/8 with five. The loss weights can thus be used to rebalance the widths.

Table T1. Five vs. nine supported widths over 512-1024 at the 119M-285M scale: parameters, validation loss (↓) and 5-shot accuracy over 13 tasks (%, ↑) of each width, better of the two in bold. Avg. is over the five shared widths.

WidthParams (M), 5 / 9 widthsVal. loss, 5 widthsVal. loss, 9 widthsAcc., 5 widthsAcc., 9 widths
512118.8 / 118.83.4063.42434.735.3
576- / 134.7-3.374-35.9
640152.7 / 151.73.3373.34235.136.2
704- / 169.7-3.317-36.4
768190.8 / 188.73.2933.29835.936.5
832- / 208.8-3.282-36.0
896233.1 / 230.03.2633.26736.736.3
960- / 252.1-3.254-35.9
1024279.6 / 275.43.2403.24336.436.2
Avg.-3.3083.31535.736.1

When the smallest required budget is much smaller than the others, we recommend a thin core followed by a wide first shell, which extends the range of budgets with almost no effect on the performance of the largest sub-network. For example, a 128-wide core followed by a 512-wide first shell supports 17.1M-279.6M parameters instead of 118.8M-279.6M (16.3× vs. 2.35×), and the validation loss of its largest sub-network is 3.241 vs. 3.240 (Table 28).

AuthorsOpenReview ↗

Official Response to Reviewer ag77 (Part 2)

W2&Q2 Performance, training efficiency and hyperparameter transfer at larger scales and with more supported widths

We acknowledge that our models are small compared with current mainstream LLMs, and a scale like that cannot be trained within the discussion period. However, ShellElastic keeps its advantage over our three scales, which span a 14× range of model size (119M-1.69B) and 7-100× the parameters of the 17.1M proxy from which all hyperparameters are transferred. At all three scales, it has lower average validation loss and higher average 5-shot accuracy than MatFormer, Fully Elastic, and Sampled Standalone. Its gap in average validation loss to the Standalone models, which in total takes five times its training steps, is also the smallest at the largest scale (Table 2).

μ\mu-Elastic also keeps the hyperparameter transfer at the largest scale and with more supported widths. We further sweep η0\eta_0 over the same seven values for the 675M-1.69B ShellElastic, whose Max has 100× the parameters of the proxy, and for the ShellElastic with nine supported widths of W1&Q1 (512-1024, every 64). The results are shown in Table T2. In both sweeps, the optimal η0\eta_0 is the base value 2×10−32\times10^{-3} or adjacent to it, as in Figure 2.

Table T2. Sweeps of η0\eta_0 with all other hyperparameters transferred from the proxy: average validation loss (↓) minus the best of each row, best in bold. Each run uses the recipe of Table 2 at its scale. At 675M-1.69B, every value of η0\eta_0 is divided by 2\sqrt{2} for the longer training, as Appendix F.1 does for the base value.

ScaleParams (× proxy)Supported widths10−410^{-4}3.3×10−43.3\times10^{-4}10−310^{-3}2×10−32\times10^{-3} (base)3.3×10−33.3\times10^{-3}10−210^{-2}3.3×10−23.3\times10^{-2}
675M-1.69B39-100×5+0.139+0.0340+0.005+0.008+0.032+0.059
119M-285M7-16×9+0.218+0.058+0.0070+0.002+0.026+0.072

The training efficiency of ShellElastic improves with scale and holds with more supported widths. This is because one full-width forward pass trains all supported widths, and the only extra cost of each width is its readout. With a shared readout and incremental logits (see our response to Reviewer ZAhc, W2), the logits of all widths cost the same FLOPs as those of Max alone, so the FLOPs per step do not increase with the number of widths. A sample-based method such as MatFormer or Fully Elastic instead trains one sampled width per step, so the steps it needs to train every width as often increase linearly with the number of widths.

ShellElastic is also more efficient than the baselines in both training and inference. At matched training FLOPs, it still outperforms MatFormer, Fully Elastic, and Sampled Standalone in average validation loss and 5-shot accuracy, and it has the highest GPU utilization (40.5% vs. 36.1% MFU on A100 GPUs; see our response to Reviewer ZAhc, W2 and Q2). At inference, a smaller width can continue decoding from the KV cache of Max exactly, as its KV cache is a slice of that of Max, whereas the baselines must re-encode the context and keep a second cache. The smaller widths can thus draft tokens for Max in speculative decoding without extra KV cache, and ShellElastic has the highest speedup at batch size 16 at both scales (e.g., 1.69× vs. at most 1.56× at 675M-1.69B with greedy decoding; see our response to Reviewer ZAhc, Q1).

Thank you for your responses. I maintain my score from the positive side.

Reviewer_ZAhcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ShellElastic, a transformer architecture that supports elastic training in a single forward pass. This is achieved by parametrizing weight matrices as block lower triangular. Matrices are composed of a core and a set of expansion shells, where each (core, shell) reads its associated dimensions and the ones from smaller groups, and only writes to its output dimensions. Furthermore, they introduce group-local normalization, which uses separate statistics and gains for the hidden state's core and its shells.

The authors further introduce a μ\mu-P style parametrization for elastic models, named μ\mu-Elastic, and show that such parametrization allows for hyperparameter transfer from a non-elastic proxy model to elastic models of various sizes, for their method and two elastic baselines.

Compared to other elastic baselines under the same number of steps, ShellElastic models generally have matched or lower validation loss and higher 5-shot accuracy across all subnetworks.

Critical strengths
  1. The paper is well written and easy to read. Figures are clear, and make the method easy to understand.
  2. The proposed method tackles several challenges of previous elastic training approaches, e.g., keeping a fixed per-step workload and guaranteeing that each sub-network is updated during each forward pass, at a cost approximately equivalent to a single forward, excluding readouts.
  3. The authors propose a μ\mu-P style parametrization for elastic models, and show that such parametrization works well for their approach and the baselines, i.e., MatFormer and a transformer adaptation of Slimmable Networks.
Critical weaknesses
  1. The core and shells are fixed once before training. Intermediate dimensions cannot be deployed, contrary to MatFormer and slimmable networks.
  2. Table 2 does not seem to be matched in terms of optimization FLOPs. Each forward step of ShellElastic is approximately equivalent to a full model forward, while a MatFormer forward, assuming uniform sampling between smallest and larger model, should be approximately ~70% of it. This also excludes the readouts over the vocabulary, which should require significant additional compute at the smaller sizes. It would be interesting to see FLOP-matched performance.
  3. The weight matrices of ShellElastic models are partially sparse, and, to my understanding, shell-based expansion significantly increases the width of the network compared to e.g., MatFormer. This should then reflect in terms of a bigger KV-cache size for the large variants, which might be important on e.g., devices with limited memory.
Questions for authors
  1. Would it be possible to alternate models while decoding the same sequence? Say, start decoding with the largest model, and switch to a smaller shell mid-decoding? What would be the impact on performance?
    1. Related, would smaller nested models be good drafters for the largest model? Recent research [1] has found that to be the case. It would be interesting if the same findings applies to ShellElastic models.
  2. Have you measured the performance of ShellElastic in a realistic training setup, e.g., in terms of device utilization?
  3. Have you tried training ShellElastic ViTs?

[1] Matryoshka Language Model Suites, N. Godey & Y. Artzi, arXiv, 2026

5 comments
AuthorsOpenReview ↗

Official Response to Reviewer ZAhc (Part 1)

We thank Reviewer ZAhc for the insightful comments and suggestions, and for the overall positive assessment of our paper. We appreciate that you find the paper well written and easy to read, that ShellElastic tackles several challenges of previous elastic training approaches, and that μ\mu-Elastic works well for both our architecture and the baselines. We address your concerns point by point below.

W1 Width fixed before training

We acknowledge that the supported widths S\mathcal{S} must be chosen before training. This is because exact reuse requires that, for each supported width ww, the first ww hidden dimensions do not depend on the rest in any layer, which ShellElastic builds into its weights and normalization. In return, one full-width forward pass produces the predictions of all supported sub-networks with a fixed workload at every step, whereas sample-based methods, such as MatFormer, Fully Elastic and Sampled Standalone, need one forward pass for each sub-network and train only the sampled sub-network at each step.

However, S\mathcal{S} is a free design choice, so the intermediate widths needed for deployment can be added to S\mathcal{S} before training. ShellElastic still trains every supported width on every token, however many widths are supported. A sample-based method instead trains each width on only 1/∣S∣1/|\mathcal{S}| of the steps on average, so each width is trained on fewer tokens as the number of widths grows, and its loss follows how often it is trained (Appendix K(1)). To train every width as often as ShellElastic does, a sample-based method needs ∣S∣|\mathcal{S}| times the steps. Wider widths can also be added after training: shells can be appended with all trained weights frozen, and every trained sub-network stays an exact prefix of the extended model (Appendix K(9)).

The original slimmable networks [1] also run only at their trained widths. MatFormer's Mix'n'Match can give intermediate model sizes, but MatFormer makes only the FFN elastic while keeping the residual dimension fixed, so the parameter ratio between its largest and smallest sub-networks has a ceiling.

[1] Slimmable neural networks. ICLR 2019

W2 FLOPs-match performance.

Excluding the readouts, one ShellElastic training step in our experiments trains all five sub-networks for 1.3-1.35× the FLOPs of one baseline step. The readouts indeed take a large share of the FLOPs, especially when the model is small. However, ShellElastic can use one shared readout for all sub-networks and compute the logits incrementally: the partial logits of each shell are computed once and accumulated from the core outwards, so computing the logits of all sub-networks takes the same FLOPs as computing the full-width logits once, and a step costs about 1.4× a baseline step. This approach brings ShellElastic another advantage: for a given largest width, its FLOPs per step do not increase with the number of supported widths, while every step still trains all of them. A sample-based baseline instead trains one sampled width per step, so the cost of training every width as often as ShellElastic does increases linearly with the number of widths.

We train ShellElastic for 25k steps and give each baseline the same FLOPs by training it for 1.4× the steps (35k steps), with other hyperparameters kept unchanged. The results are shown in Table T1. At matched FLOPs, ShellElastic is still better than the baselines: it has the lowest average validation loss, with the largest gain at the largest width, and the best or tied-best 5-shot accuracy at every width. At the same 25k steps, independent readouts give 0.002-0.004 lower loss than the shared readout at every width but cost about 1.5× the FLOPs. We thus recommend using the shared readout with incremental logits when matched-FLOPs performance matters and data is sufficient, and independent readouts for the best matched-steps performance (Table 2) when data is limited.

Table T1. FLOPs-matched performance at the 119M-285M scale: validation loss (↓, left) and 5-shot accuracy over 13 tasks (%, ↑, right) of each sub-network, best or tied-best at matched FLOPs in bold.

MethodFLOPsStepsLowMediumHighExtraMaxAvg.LowMediumHighExtraMaxAvg.
Sampled Standalone10.94e1835k3.5883.5463.4833.4573.4413.50333.834.334.635.434.034.4
Fully Elastic10.94e1835k3.4203.3453.3033.2833.2773.32634.835.336.135.936.235.7
MatFormer10.70e1835k3.4073.3303.2953.2803.2763.31734.935.535.736.036.035.6
ShellElastic (Shared Readout)10.99e1825k3.4133.3403.2983.2683.2443.31335.435.836.436.036.936.1
ShellElastic (Independent Readout)16.58e1825k3.4093.3383.2943.2643.2403.30935.536.636.636.936.936.5
AuthorsOpenReview ↗

Official Response to Reviewer ZAhc (Part 2)

W3 KV cache of the larger sub-networks

We acknowledge that in our experiments, the larger sub-networks of ShellElastic has a larger KV cache than those of the baselines. However, the size of the KV cache is set by the number of key and value heads, which ShellElastic does not fix: it supports grouped-query attention (GQA) [2], in which several query heads share one key and value head in the same width group in ShellElastic. In each block of g=128g = 128 dimensions, we use two 64-dimensional query heads that share one 64-dimensional key and value head instead of one 128-dimensional head, which halves the KV cache of every sub-network (64-dimensional heads alone have a small effect on the loss, Appendix K(11)). Each key and value head is computed by the same block as the two query heads that read it, so Proposition 1 still holds, and the KV cache of each width is still a slice of that of Max (see Q1). We train this model with the recipe and HPs of Table 2 without re-tuning, as μ\mu-Elastic sets the factors of every weight from its fan-in and the FFN expansion.

The results are shown in Table T2. With GQA, ShellElastic caches 32-64 KiB per token, no more than either baseline at any width (0.50-1.00× MatFormer's and 0.67-0.80× Fully Elastic's KV cache), and it is still better than both baselines at every width: its average validation loss is 3.314 vs. 3.336 for Fully Elastic and 3.347 for MatFormer, with the largest gain at the largest width (3.244 vs. 3.290 and 3.314), and its 5-shot accuracy is higher at every width (average 36.7 vs. 35.6 for both). Compared with multi-head attention (MHA), the configuration of Table 2, GQA raises the validation loss by 0.004-0.006 nats per width (3.314 vs. 3.309 on average), does not lower the 5-shot accuracy (36.7 vs. 36.5), and halves the KV cache.

Table T2. Grouped-query attention (GQA) at the 119M-285M scale: parameters and KV cache per token (bf16) from the smallest to the largest sub-network, and validation loss (↓, left) and 5-shot accuracy over 13 tasks (%, ↑, right) of each sub-network, best of the baselines and ShellElastic (GQA) in bold. All models are trained for 25k steps with the recipe of Table 2; Fully Elastic, MatFormer, and ShellElastic (MHA) are from Tables 2 and 9.

MethodParams (M)KV cache (KiB/token)LowMediumHighExtraMaxAvg.LowMediumHighExtraMaxAvg.
Fully Elastic119.0-287.548-803.4293.3543.3153.2953.2903.33634.935.535.735.936.035.6
MatFormer118.8-282.4643.4253.3573.3253.3123.3143.34735.135.235.835.935.935.6
ShellElastic (GQA)118.3-278.232-643.4153.3433.3003.2693.2443.31435.937.236.836.936.936.7
ShellElastic (MHA)118.8-279.664-1283.4093.3383.2943.2643.2403.30935.536.636.636.936.936.5

On devices with limited memory, the smaller sub-networks of ShellElastic with GQA need only 32-56 KiB per token, whereas every sub-network of MatFormer needs 64 KiB, since MatFormer keeps its attention at full width. The KV cache of Max, 64 KiB per token, still contains the KV caches of all smaller widths, so the exact switch to a smaller width (see Q1) needs no second cache.

[2] GQA: Training generalized multi-query transformer models from multi-head checkpoints. EMNLP 2023

Q1 Switching widths while decoding a sequence; smaller widths as drafters

ShellElastic can switch from Max to any smaller supported width at any token, exactly and without recomputation. By Proposition 1, the hidden states of width ss are the first sgsg dimensions of those of Max in every layer, so its KV cache is a slice of the KV cache of Max. After the switch, width ss continues from this slice with the same predictions as if it had read the whole sequence itself. MatFormer and Fully Elastic do not nest hidden states, so an exact switch must re-encode the whole context with the smaller width and keep a second cache.

We measure the impact with the final checkpoints of Table 2 in bf16 on 256 C4 validation sequences of 2,048 tokens. Max encodes tokens 1-1,024, and Low then predicts tokens 1,025-2,048 from its own KV cache or from that of Max. The results are shown in Table T3. The switch does not change the loss of ShellElastic, and its 0.4-0.5% of differing top-1 predictions come from bf16 rounding (in fp32, all are the same). For MatFormer and Fully Elastic, reusing the cache of Max raises the loss of Low and changes 3-5% of its top-1 predictions.

AuthorsOpenReview ↗

Official Response to Reviewer ZAhc (Part 3)

Table T3. Switching from Max to Low after 1,024 tokens: validation loss (↓) of Low on tokens 1,025-2,048 with its own KV cache and with that of Max, and the share of positions with the same top-1 prediction.

ScaleMethodOwn cacheMax cacheChangeSame top-1 (%)
119M-285MFully Elastic3.3463.347+0.000696.9
MatFormer3.3463.353+0.006795.7
ShellElastic3.3383.3380.000099.5
675M-1.69BFully Elastic2.8732.875+0.001496.9
MatFormer2.8462.861+0.015594.8
ShellElastic2.8472.8470.000099.6

The smaller widths can also serve as draft models of the wider ones in speculative decoding [3], without extra weights, KV cache, training or prefill. The weights of a smaller width are the first part of those of a wider one, and by Proposition 1, so is its KV cache. The smaller width thus drafts tokens from the cache of the wider one, and the wider one verifies them in one forward pass, which also writes their full keys and values into the cache.

We run speculative decoding with the final checkpoints of Table 2 on one A100 GPU. We generate 512 tokens for each of 32 C4 validation prompts of 1,024 tokens, with γ=4\gamma = 4 drafted tokens per round, greedy decoding and nucleus sampling (temperature 0.4, top-p 0.9), and batch sizes 1 and 16. In each elastic model, Low drafts for Max, and in Standalone, the separately trained Low drafts for the separately trained Max. We report the speedup predicted from the bytes of weights and KV cache read per step [3]. The results are shown in Table T4. Nested sub-networks are good drafters, and they agree with their Max much more often than the separately trained Low on held-out text. MatFormer and Fully Elastic draft as well as ShellElastic, but their Low needs a KV cache of its own, which costs a second prefill and extra memory. ShellElastic also has the highest speedup at batch 16 at both scales. This is because reading the KV cache dominates at batch 16, and the Low of ShellElastic reads only half of the cache of Max, whereas the Low of MatFormer reads a cache as large as that of Max. Moreover, ShellElastic can raise the agreement between Low and Max by in-place distillation without an extra forward pass, as one forward pass of Max gives the predictions of all sub-networks, whereas MatFormer and Fully Elastic, which train one sampled width per step, need an extra forward pass of Max to compute the distillation loss.

Table T4. Low drafting for Max at γ=4\gamma = 4: extra KV cache of the draft (↓), top-1 agreement on held-out text (↑), tokens per round (↑, at most 5) at batch 1, and speedup (↑) over decoding with its Max alone at batch 1 and 16. Tokens per round and speedups are for greedy decoding / nucleus sampling, best per scale in bold.

ScaleMethodExtra KV (KiB/token)Agreement (%)Tokens per roundSpeedup, batch 1Speedup, batch 16
119M-285MStandalone4874.34.73 / 4.601.75 / 1.701.48 / 1.40
Fully Elastic4879.84.79 / 4.651.77 / 1.721.49 / 1.46
MatFormer6480.34.79 / 4.681.71 / 1.671.12 / 1.10
ShellElastic080.74.79 / 4.681.76 / 1.711.63 / 1.59
675M-1.69BStandalone10477.34.67 / 4.401.78 / 1.671.53 / 1.49
Fully Elastic10483.14.77 / 4.631.81 / 1.761.56 / 1.50
MatFormer12884.64.80 / 4.631.83 / 1.761.32 / 1.29
ShellElastic084.04.79 / 4.721.85 / 1.821.69 / 1.64

[3] Fast inference from transformers via speculative decoding. ICML 2023

Q2 Device utilization in real training

The latency and peak training memory of the 119M-285M models are shown in Table 10 of the paper. We further measure the device utilization of ShellElastic and the elastic baselines during training on 8 A100-40GB GPUs and 8 H100-80GB GPUs, in terms of MFU [4], HFU, and the GPU-Util reported by nvidia-smi. The results are shown in Table T5. All methods keep the GPUs busy (GPU-Util 91-96% on A100 and 84-86% on H100), and ShellElastic has the highest MFU, HFU, and GPU-Util on both GPUs. This is because every step of ShellElastic runs the same full-width workload that trains all sub-networks, whereas each baseline step trains one sampled sub-network, and many weight parameters are not used during smaller sub-network training steps, and the smaller sub-networks use the GPUs less efficiently.

Table T5. Device utilization (%, ↑) during training at the 119M-285M scale on 8 A100-40GB GPUs (left) and 8 H100-80GB GPUs (right), best in bold.

MethodMFUHFUGPU-UtilMFUHFUGPU-Util
Fully Elastic36.137.891.517.017.884.0
MatFormer36.138.991.216.617.485.0
ShellElastic40.543.896.117.919.386.0

[4] PaLM: Scaling language modeling with pathways. JMLR 2023

Reviewer_NDQ1OpenReview ↗
Rating
3
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes an elastic pre-training method that shows three properties: Scalability, Transferability, and Simultaneous maximal updates. This is achieved through what they call the SHELLELASTIC architecture and the μ-ELASTIC parameterization. The authors provide theoretical guarantees of certain properties along with empirical evaluations.

Critical strengths

Elastic training is a timely topic, and having theoretical guarantees is good.

Critical weaknesses

I am not an expert on this paper. I read it carefully, and found no critical weakness.

Questions for authors

NA.

1 comment
AuthorsOpenReview ↗

Official Response to Reviewer NDQ1

We thank Reviewer NDQ1 for the time and effort devoted to reviewing our paper, and for the overall positive assessment of our work. We appreciate that you find elastic training a timely topic and our theoretical guarantees valuable, and that you found no critical weakness in the paper. We welcome any comments and suggestions you may have at any time, and would be glad to discuss with you how to further strengthen our contributions and improve ShellElastic.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Highlighted General Response

We thank all reviewers for the insightful comments and suggestions, and for the overall positive assessment of our paper. We also thank the Area Chairs, Senior Area Chairs and Program Chairs for their time and effort. We appreciate that the reviewers find elastic training a timely topic (NDQ1) and the paper well written (ZAhc), that ShellElastic achieves exact computation reuse and tackles several challenges of previous elastic training approaches (ZAhc, ag77), that μ\mu-Elastic provides hyperparameter scaling rules with theoretical guarantees and works well for both our architecture and the baselines (NDQ1, ZAhc, ag77), and that our experiments provide substantial empirical support for our designs (ag77).

We address the reviewers' concerns and questions with new experiments and summarize the main results below, with the details in our response to each reviewer. Unless stated otherwise, all results are at the 119M-285M scale.

1. Matched FLOPs and training efficiency (ZAhc W2 and Q2). With a shared readout, a ShellElastic step trains all five sub-networks for about 1.4× the FLOPs of a baseline step, which trains a single sub-network. We train the baselines for 1.4× the steps to match FLOPs, and ShellElastic still has the lowest average validation loss and the highest average 5-shot accuracy. In training on A100 and H100 GPUs, it also has the highest MFU, HFU and GPU-Util, as every step runs the same full-width workload.

2. KV cache and decoding (ZAhc W3 and Q1). We train ShellElastic with grouped-query attention within each width group, which halves the KV cache of every sub-network. With it, ShellElastic caches no more than either baseline at any width (32-64 KiB per token) and still outperforms both at every width without re-tuning the hyperparameters. As the KV cache of each width is a slice of that of the largest width, decoding can also switch from the largest to any smaller width at any token, exactly and without recomputation. At both 119M-285M and 675M-1.69B, this switch leaves the loss unchanged, whereas reusing the cache of the largest width changes 3-5% of the top-1 predictions of MatFormer and Fully Elastic. For the same reason, the smaller widths draft for the largest one in speculative decoding without an extra KV cache, and ShellElastic has the highest speedup at batch size 16 at both scales.

3. Choice of supported widths (ZAhc W1, ag77 W1&Q1). The supported widths must be chosen before training, but any widths needed for deployment can be included, and wider widths can be appended after training with all trained weights frozen. As ShellElastic trains every width at every step, an added width costs the other widths much less than in sample-based methods, which train each width less often. With a shared readout, an added width also adds no FLOPs per step. We train ShellElastic with nine instead of five supported widths over 512-1024, and the five shared widths keep their 5-shot accuracy at only 0.007 nats higher average validation loss. In our architecture search, a large core with thin shells has the lowest validation loss, or is within 0.006 nats of it, at every width. When a much smaller budget is required, a thin core followed by a wide first shell raises the parameter ratio of the largest to the smallest sub-network from 2.35× to 16.3× with almost no change in the loss of the largest one.

4. Scale and hyperparameter transfer (ag77 W2&Q2). The paper verifies the hyperparameter transfer from the 17.1M proxy at the two smaller scales (Figure 2). We further sweep the base learning rate η0\eta_0 at the largest scale, 675M-1.69B, whose largest sub-network has 100× the parameters of the proxy, and with nine supported widths. In both sweeps, the value transferred from the proxy is still near optimal.

5. Vision transformers (ZAhc Q3). We are training ShellElastic diffusion transformers (150M-375M parameters), which apply the ViT architecture to latent image patches, and will post the results once they finish.

We will add all new results and discussions to the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.