- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies computation reuse in language models with variable widths. Submodels of different widths can produce different intermediate representations even when they share parameters, requiring recomputation when the model is widened. The authors introduce prefix invariance and enforce block-triangular mappings, together with hierarchical normalization, attention head grouping, and a cumulative output projection. These changes make the hidden states of smaller models exact prefixes of those of larger models, allowing existing computation and KV cache entries to be reused. The paper also introduces two training techniques, HSD and MNG, and strategies for handling missing historical KV entries. Experiments use approximately 1B-parameter Llama and Nemotron architectures, with large-batch decoding throughput evaluated on a single H200 GPU.
Critical strengths
Prefix invariance captures a key requirement for reusing computation as model width increases. Sections 2 and 3.1 clearly explain why parameter sharing alone is insufficient and describe the conditions that each Transformer component must satisfy. In particular, the changes to normalization and attention keep existing features and KV entries valid when the model is widened. I consider this the paper's most valuable contribution.
Critical weaknesses
- HSD and MNG lack ablation results on language models. Section 3.2 introduces these two training techniques, and Appendix D.1 states that both are used in the main language model experiments. However, the HSD comparison in Figure 3 and Appendix D.2 is conducted on low-rank linear regression. I could not find language model results with HSD or MNG removed. This makes it difficult to assess which model widths benefit from each technique and by how much. Such a comparison would help clarify the role of these training techniques.
- The presentation of the experimental results makes the performance gains difficult to assess. Some axis labels and legends in Figures 6 and 7 are too small to read comfortably at normal viewing size. The discussion of Figures 5 and 6 is also mostly qualitative, leaving readers to estimate the gains over the baselines from the curves. I would like the authors to select a few representative settings and report the accuracy improvement at the same compute budget, or the computation saved at comparable prediction agreement levels. Appendix Table 3 already provides throughput speedup figures, but the main text offers little quantitative comparison or discussion of these results.
Questions for authors
- Are language model ablation results available for removing HSD, replacing HSD with distillation from the full model, and removing MNG?
- With the training data and nested widths held fixed, how does each setting affect the quality of the submodels and the full model?
2 comments
Dear Reviewer,
Thank you again for your feedback. As the discussion deadline approaches, we would appreciate hearing whether our response and revised manuscript address your concerns. We hope you will consider raising your score in light of the revisions. We are happy to clarify any point that needs further explanation.
Response to W1, Q1 & Q2 (Language Model Ablations for HSD and MNG):
Thank you for pointing this out. We agree these ablations were absent from our manuscript. We have now added them for GPT-2 (Table 2). Setup. We follow standard practice in knowledge distillation (Hinton et al., 2015) and train every variant on a weighted combination of cross-entropy on the labels and a distillation term. The two variants differ only in the teacher: either the full model (CE + KD) or the next-larger submodel (CE + HSD). The mixing weight is tuned separately for each, so that both are compared at their best setting.
HSD. HSD is not required for Prefix-LM to train, but is our choice of distillation target. Each submodel is distilled from the next larger one instead of from the full model, which costs nothing here because prefix invariance yields all K outputs in a single pass. With the cumulative classifier, the teacher terms of the HSD term cancel in the summed gradient (Sec. A.1), so it acts as stagewise label supervision: each block learns to correct the prediction of the prefix below it. With CE + HSD, the full model achieves a loss of 3.461, compared to 3.494 with CE + KD (+0.033), while the submodels are within 0.5% of CE + KD. In the SGD experiment below, HSD also converges markedly faster than KD across all widths (new Fig. 15). Finally, HSD has a systems benefit: its backward pass requires only one hard-label loss per block, which, in the same setup, yields 7% higher training throughput and 15% lower peak memory than KD.
MNG. In the language-model experiments, turning MNG off changes the loss by at most 0.12% at any width. This is expected, and we should have stated it: MNG multiplies each block's gradient by a constant, and all our language models are trained with AdamW, which normalizes every coordinate's update by the running magnitude of its own gradient. A constant per-block scale therefore cancels out, except for the effect of ε and clipping, and MNG is effectively inactive in these runs. It matters for optimizers without this invariance:
We have revised Sec. 3.2 to state the role of both techniques in these terms.
Response to W2 (Presentation Improvements & Quantitative Comparisons):
Thank you for the suggestion. We have polished the plots and discussion so that the results showcase the gains our method can achieve.
To this end, we present a single downstream-task plot (revised Figure 5) comparing the quality-compute trade-off. We report average accuracy across six different benchmarks described in Section C.2.1. We train a router for both architectures, Prefix-LM and Slimmable, and separate this comparison from CALM and LayerSkip. For example, on Llama-3.2-1B, Prefix-LM achieves 45.02% accuracy at 0.946 GFLOPs/token, compared with 44.58% at 3.846 GFLOPs/token for Slimmable cascading; on Qwen3.5-2B, Prefix-LM achieves 48.92% at 2.476/token, using estimated FLOP counts.
We have now refined the presentation of the quality-performance tradeoff, and now report speedup in Figure 6 and FLOPs comparison in Figure 7, in a matched setting. For example, Figure 7 shows that Prefix-LM with the masking strategy requires only 70.2% of the FLOPs of the fixed-width medium model to match its performance, a 1.42x theoretical compute speedup. Highlighting the efficiency of our method, we report that Prefix-LM with cascading achieves a higher MT-Bench evaluation score of 1.90, compared to 1.85, at the same latency.