- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Decoder-only language model until now allocated the same parameter budget to each layer. In this study, the authors investigate the effect of changing the total parameters per layer, i.e., increasing and decreasing the number of parameters per layer. The change in the total paramaters/layer is achieved by controlling the intermediate layer size of the FFN layer, keeping everything else the same. First, the authors show that decreasing the parameters/layer for latter layers improves the performance (hence, "tapered" LM). Based on the initial finding, the authors then scale up the experiments and note the following findings
- Cosine schedule works the best for decreasing paramaters/layer for deeper layers (lowest perplexity)
- Cosine schedule works well with multiple transformer variants (lower perplexity and higher commonsense reasoning performance, compared to fixed paramters/layer model at same FLOPs)
- Tapering reduces throughput and increases memory consumption for the larger models
- With off-the-shelf GPT models, authors show that the deeper layers only reinforce the residual direction, diminishing their contribution
Critical strengths
- The paper is very well written and organized
- The authors investigate a novel research question
- Experiments are well structured and precisely answer the research question
- All results consistently highlight the benefit of tapering with improvements in in-domain data and downstream tasks
Critical weaknesses
- The paper does not investigate why deeper layers do not add value. Section 4 shows that deeper layers only reinforce the residual direction, but the 'why' is not answered. I completely understand if answering
whyis out of scope of the study. However, it might be helpful to make this clear in the manuscript. - I would like to see more discussion on how practitioners must tackle the reduction in throughput. For example, for one GPU inference, practitioners will need to set the batch size based on the first layer, which may lead to underutilization of the GPU during deeper layers' forward pass. For a multi-GPU setup, layers could be separated, and this issue can be partially resolved. However, the earliest layer will be the throughput bottleneck; it will play a part in setting the batch size, or it will decide the rate at which the next layer receives the requests. Please correct me if I am wrong.
- The reasoning behind using older architectures (section 4: GPT-2, otherwise: llama-2) for the experiments is not clear
- Because the main control variable (FFN intermediate size) will have a direct impact on MoE architectures, the paper may benefit from including MoE models in the experiments
Questions for authors
- line-51: what is surface diversity?
- line-375: given that one variable (layer index) in the correlation test is ordinal, Spearman rho might be more suitable? Please report p-values as well.
- Please consider improving the readability of the plots. Currently, the numbers and the text on the plots are not readable.
5 comments
Part 2/2
MoE
We agree that MoE models are a natural next step, since tapering acts on the FFN, which MoE replaces with experts. However, MoE introduces several additional per-layer axes (number of experts, expert width, and number of active experts) that interact with tapering, and a controlled study would require its own experimental design. We have added this as future work in the Discussion section.
“Surface diversity”
By surface diversity, we mean the differences in the token-mixing module across model families (e.g., softmax attention, gated attention, recurrent and state-space layers, or memory modules), while the block structure itself (a token-mixing module followed by an MLP, stacked uniformly across depth) is shared. We have rephrased this sentence in the Introduction as “despite their differences in token mixing.”
Correlation statistics
Thank you, we agree. We now report Spearman's rank correlation () between layer index and the mean cosine similarity, with two-sided p-values from a permutation test (200,000 permutations), and have replaced the Pearson correlations in Section 4 accordingly (Appendix B, Table 5). The table below uses 262,144 tokens; with the 2,048 tokens of the submission, the all-layer values change by at most 0.03. As in Figure 4, the first and last layers are excluded.
| Model | MLP, all layers | MLP, second half | Block, all layers | Block, second half |
|---|---|---|---|---|
| GPT-2 124M | 0.61 (p=0.066) | 1.00 (p=0.017) | 0.65 (p=0.049) | 1.00 (p=0.017) |
| GPT-2 355M | 0.69 (p<0.001) | 0.99 (p<0.001) | 0.42 (p=0.056) | 1.00 (p<0.001) |
| GPT-2 774M | 0.35 (p=0.042) | 0.94 (p<0.001) | 0.37 (p=0.030) | 0.97 (p<0.001) |
| GPT-2 1.5B | 0.51 (p<0.001) | 0.93 (p<0.001) | 0.46 (p=0.002) | 0.98 (p<0.001) |
| Qwen2.5 0.5B | 0.51 (p=0.017) | 0.63 (p=0.044) | 0.57 (p=0.006) | 0.63 (p=0.043) |
| Qwen2.5 1.5B | 0.57 (p=0.003) | 0.92 (p<0.001) | 0.58 (p=0.002) | 0.97 (p<0.001) |
| Qwen2.5 7B | 0.25 (p=0.22) | 0.80 (p=0.002) | 0.25 (p=0.22) | 0.86 (p<0.001) |
| Qwen2.5 14B | 0.27 (p=0.067) | 0.78 (p<0.001) | 0.32 (p=0.028) | 0.84 (p<0.001) |
Over all layers, is positive for every model and both quantities, and significant (p < 0.05) in 11 of 16 cases. The all-layer correlation understates the trend because the curves are not monotone: alignment first drops over the early layers and then rises steadily through the second half of the network (Figure 4), as described in Section 4. Restricted to the second half, ranges from 0.63 to 1.00 with p < 0.05 for every model and both quantities. We report both in Appendix B (Table 5).
Figure readability
Thank you. The new figures in Appendix B use larger, full-width panels, and we will increase the size of the text and numbers in the remaining figures for the final version.
Follow-up
Dear Reviewer, as the discussion period closes tomorrow, we wanted to kindly follow up on our response. Based on your suggestions, we added new experiments and analyses that we believe have meaningfully strengthened the paper, and we would appreciate your feedback on whether they address your concerns.
Part 1/2
We thank the reviewer for their thoughtful feedback. Changes made in response are highlighted in light blue in the updated PDF (we also refer to them below).
We agree that Section 4 characterizes what happens (later MLP updates become increasingly aligned with the residual stream) rather than why. While we also believe a full answer to this is beyond the scope of this study, one candidate explanation comes from the “curse of depth” phenomenon (Sun et al., 2025). Modern language models use Pre-LN, because normalizing the residual stream itself (Post-LN) hampers gradient flow, which makes deep models difficult to train. In Pre-LN models, each layer’s output is added to an unnormalized residual stream, so the variance of the residual stream grows with depth. As a result, each layer’s update becomes small relative to the residual stream, and deeper layers increasingly behave like identity mappings. This account concerns the magnitude of layer updates, while our analysis measures their direction, so the two are complementary. Both point to the same practical conclusion: since later layers contribute less, it matters to allocate capacity across depth more carefully. We have added this discussion, including the scope of our analysis, to the end of Section 4 (highlighted in blue).
Thank you for raising this. We would first like to clarify that the throughput reduction in Table 3 is measured during training. As tapering keeps total FLOPs identical, this reduction does not come from additional computation; we expect most of it comes from the implementation. Our standard training implementation (e.g., fused kernels and their tuned configurations) is optimized for a stack of identical layers, and layers of different shapes break this assumption. A practical fix is to select kernel configurations separately for each layer width and to round widths to multiples of larger tile sizes (e.g., 64 or 128 instead of 16), so that every layer’s matrix multiplications run efficiently. As the reduction was small in our experiments (at most 6%), we did not optimize kernels for tapered models in this work. At larger scales, however, this cost matters more, and it can be addressed without changing the model. These fixes apply to both training and inference.
Regarding single-GPU inference, we would also like to clarify that the batch size is not set by the first layer. Tapering keeps total parameters and FLOPs fixed and does not change the attention layers, so the model weights and the KV cache, which dominate memory at inference, are unchanged. Moreover, since layers run one after another, the narrower later layers do not leave the GPU idle; they simply finish faster, and any efficiency cost from their smaller matrix multiplications can be reduced with the optimized kernels described above.
For multi-GPU settings, tensor parallelism (Shoeybi et al., 2019), widely used in both training and serving, splits each MLP’s intermediate dimension across GPUs, so every GPU processes every layer and no layer becomes a bottleneck. If layers are instead split across GPUs (pipeline parallelism), the reviewer is right that an even split would make the first stage the bottleneck. Because total compute is unchanged, this can be avoided in both training and inference by partitioning layers according to their FLOPs rather than their number, for example by assigning fewer of the wide early layers to the first stage.
We have added this practical guidance for both training and inference to Section 3.4 (highlighted in blue).
Llama-2: only the tokenizer comes from Llama-2, as part of the standard recipe used in recent architecture studies (Titans, Gated DeltaNet). The models themselves are recent: Transformer++ (a Llama-style architecture with RoPE, SwiGLU, and RMSNorm), Gated Attention (Qiu et al., 2025), and Titans (Behrouz et al., 2024). We have made this clearer in Section 3.1.
GPT-2: we chose it for the analysis in Section 4 because it is a controlled family with publicly available pretrained checkpoints at four sizes (124M to 1.5B) that share the same architecture and training data. To show that the pattern is not specific to GPT-2, we extended this analysis to a modern open model family, Qwen2.5 (0.5B, 1.5B, 7B, 14B), and found the same trend: MLP outputs become progressively more aligned with the residual stream in the second half of the network (Appendix B, Figure 6, shown next to GPT-2 in Figure 5; Appendix B is highlighted in light purple, as it also addresses Reviewer PBFF). We also repeated the analysis for both families with 128× more tokens (262,144 instead of 2,048 WikiText-2 tokens), and the curves and correlations are essentially unchanged (Appendix B, Figures 7 and 8).