- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ShellElastic, an elastic Transformer pretraining framework designed to support different inference budgets. ShellElastic organizes hidden dimensions into a dense core and successive shells, using block lower triangular projections to enable exact hidden-state reuse across all supported sub-networks within a single full-width forward pass. The authors further introduce μ-Elastic, an extension of maximal update parameterization (μP) to elastic models, enabling simultaneous maximal updates and hyperparameter transfer. Experiments across three model scales show that ShellElastic outperforms MatFormer and fully elastic baselines in validation loss and downstream performance, while approaching the performance of separately trained models.
Critical strengths
-
ShellElastic achieves exact computation reuse across sub-networks through block lower triangular projections and group-local normalization. μ-Elastic further provides hyperparameter scaling rules for shared parameters, supported by theoretical analysis of simultaneous maximal updates and hyperparameter transfer.
-
The paper evaluates the method across three model scales and compares it with several elastic training baselines. The hyperparameter transfer experiments, architecture search, and comprehensive ablation studies provide substantial empirical support for the proposed designs.
Critical weaknesses
-
ShellElastic achieves strong performance through a relatively large dense core and a specific configuration of supported widths. However, this design may limit its flexibility in accommodating different inference budgets. Although the paper includes architecture search experiments, the trade-off between model performance and the range of supported inference budgets remains insufficiently discussed, which may affect its practical applicability.
-
Although the paper demonstrates promising results across multiple model scales, the experiments are still limited to relatively small model sizes and a restricted set of architectural configurations. Whether ShellElastic and μ-Elastic can consistently maintain their performance, training efficiency, and hyperparameter transfer advantages at larger scales and with more supported widths remains to be established.
Questions for authors
-
Can the authors further discuss how the choice of dense core size and supported widths affects the trade-off between model performance and elasticity, and how this trade-off can be balanced under different inference budgets?
-
How are the performance and training efficiency of ShellElastic and μ-Elastic expected to scale with larger model sizes and an increasing number of supported widths?
3 comments
Official Response to Reviewer ag77 (Part 2)
W2&Q2 Performance, training efficiency and hyperparameter transfer at larger scales and with more supported widths
We acknowledge that our models are small compared with current mainstream LLMs, and a scale like that cannot be trained within the discussion period. However, ShellElastic keeps its advantage over our three scales, which span a 14× range of model size (119M-1.69B) and 7-100× the parameters of the 17.1M proxy from which all hyperparameters are transferred. At all three scales, it has lower average validation loss and higher average 5-shot accuracy than MatFormer, Fully Elastic, and Sampled Standalone. Its gap in average validation loss to the Standalone models, which in total takes five times its training steps, is also the smallest at the largest scale (Table 2).
-Elastic also keeps the hyperparameter transfer at the largest scale and with more supported widths. We further sweep over the same seven values for the 675M-1.69B ShellElastic, whose Max has 100× the parameters of the proxy, and for the ShellElastic with nine supported widths of W1&Q1 (512-1024, every 64). The results are shown in Table T2. In both sweeps, the optimal is the base value or adjacent to it, as in Figure 2.
Table T2. Sweeps of with all other hyperparameters transferred from the proxy: average validation loss (↓) minus the best of each row, best in bold. Each run uses the recipe of Table 2 at its scale. At 675M-1.69B, every value of is divided by for the longer training, as Appendix F.1 does for the base value.
| Scale | Params (× proxy) | Supported widths | (base) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| 675M-1.69B | 39-100× | 5 | +0.139 | +0.034 | 0 | +0.005 | +0.008 | +0.032 | +0.059 |
| 119M-285M | 7-16× | 9 | +0.218 | +0.058 | +0.007 | 0 | +0.002 | +0.026 | +0.072 |
The training efficiency of ShellElastic improves with scale and holds with more supported widths. This is because one full-width forward pass trains all supported widths, and the only extra cost of each width is its readout. With a shared readout and incremental logits (see our response to Reviewer ZAhc, W2), the logits of all widths cost the same FLOPs as those of Max alone, so the FLOPs per step do not increase with the number of widths. A sample-based method such as MatFormer or Fully Elastic instead trains one sampled width per step, so the steps it needs to train every width as often increase linearly with the number of widths.
ShellElastic is also more efficient than the baselines in both training and inference. At matched training FLOPs, it still outperforms MatFormer, Fully Elastic, and Sampled Standalone in average validation loss and 5-shot accuracy, and it has the highest GPU utilization (40.5% vs. 36.1% MFU on A100 GPUs; see our response to Reviewer ZAhc, W2 and Q2). At inference, a smaller width can continue decoding from the KV cache of Max exactly, as its KV cache is a slice of that of Max, whereas the baselines must re-encode the context and keep a second cache. The smaller widths can thus draft tokens for Max in speculative decoding without extra KV cache, and ShellElastic has the highest speedup at batch size 16 at both scales (e.g., 1.69× vs. at most 1.56× at 675M-1.69B with greedy decoding; see our response to Reviewer ZAhc, Q1).
Thank you for your responses. I maintain my score from the positive side.
Official Response to Reviewer ag77 (Part 1)
We thank Reviewer ag77 for the insightful comments and suggestions, and for the overall positive assessment of our paper. We appreciate that you find that ShellElastic achieves exact computation reuse across sub-networks, that μ-Elastic provides hyperparameter scaling rules for shared parameters with theoretical analysis of simultaneous maximal updates and hyperparameter transfer, and that our evaluation, hyperparameter transfer experiments, architecture search and comprehensive ablation studies provide substantial empirical support for our designs. We address your concerns point by point below.
The core size and the supported widths indeed trade off performance against elasticity, and they are not fixed to the configuration in the paper (a 512-wide core followed by four 128-wide shells). In practice, we first set the total parameters and the number of supported widths according to the budgets required for deployment. The practical design principle is then to put more parameters into the core, as this gives the best overall performance. Our architecture search keeps the total parameters and the number of widths fixed and only changes how the hidden dimensions are split between the core and the shells (Section 4.3 and Appendix J.2). A large core with thin shells has the lowest validation loss, or is within 0.006 nats of it, at every width, whereas a wide shell makes the smallest width that includes it 0.025-0.052 nats worse.
Every added width costs the other widths some performance, and this cost is much smaller for ShellElastic than for sample-based methods such as MatFormer and Fully Elastic. This is because the loss of each sub-network follows how often it is trained and its loss weight (Appendix K(1) and K(2)). A sample-based method trains one sampled width per step, so every added width makes each width trained less often. ShellElastic instead trains all widths at every step, so an added width only reduces the loss weight of the others. With a shared readout, an added width also adds no FLOPs per step (see W2&Q2).
We train ShellElastic with nine widths from 512 to 1024 in steps of 64 and compare it with five widths over the same range in steps of 128, both with a base width of 64 (Appendix K(11)) and the hyperparameters transferred from the proxy. The results are shown in Table T1. The four added widths lie between their neighbours in validation loss. The five shared widths keep their 5-shot accuracy (36.1 vs. 35.7 on average), and their validation loss rises by only 0.007 nats on average, mostly at width 512. This is because each smaller width gets 1/16 of the loss weight with nine widths instead of 1/8 with five. The loss weights can thus be used to rebalance the widths.
Table T1. Five vs. nine supported widths over 512-1024 at the 119M-285M scale: parameters, validation loss (↓) and 5-shot accuracy over 13 tasks (%, ↑) of each width, better of the two in bold. Avg. is over the five shared widths.
When the smallest required budget is much smaller than the others, we recommend a thin core followed by a wide first shell, which extends the range of budgets with almost no effect on the performance of the largest sub-network. For example, a 128-wide core followed by a 512-wide first shell supports 17.1M-279.6M parameters instead of 118.8M-279.6M (16.3× vs. 2.35×), and the validation loss of its largest sub-network is 3.241 vs. 3.240 (Table 28).