acceptodds
Under review as a conference paper at ICLR 2027

Tapered Language Models

Abstract

Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly across depth. This default is inherited from the original transformer and has remained largely unchanged, yet a growing body of evidence suggests that layers contribute non-uniformly to the final output. We ask whether parameter capacity should reflect this asymmetry. In a controlled experiment at a fixed parameter budget, allocating more MLP capacity to earlier layers and less to later layers improves perplexity over a uniform-width baseline, while the reverse allocation, the direction taken by prior layer-wise scaling schemes, hurts. Building on this result, we propose *Tapered Language Models* (TLMs), in which MLP width decreases monotonically across depth under a fixed total budget. We focus on MLPs because they dominate parameter count across modern LM families and expose width as a single, clean axis of variation. Using a smooth cosine schedule selected once on a M Transformer and transferred unchanged, tapering improves average downstream accuracy and perplexity over matched uniform baselines in all six comparisons across three architectures (Transformer, Gated Attention, and Titans) at M and B parameters, with identical parameter count and FLOPs. These results suggest depth-aware capacity allocation as a simple, architecture-agnostic guideline for language model design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient Language Models, Depth-Aware Capacity Allocation, MLP Width Tapering

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_K1o8OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Decoder-only language model until now allocated the same parameter budget to each layer. In this study, the authors investigate the effect of changing the total parameters per layer, i.e., increasing and decreasing the number of parameters per layer. The change in the total paramaters/layer is achieved by controlling the intermediate layer size of the FFN layer, keeping everything else the same. First, the authors show that decreasing the parameters/layer for latter layers improves the performance (hence, "tapered" LM). Based on the initial finding, the authors then scale up the experiments and note the following findings

  • Cosine schedule works the best for decreasing paramaters/layer for deeper layers (lowest perplexity)
  • Cosine schedule works well with multiple transformer variants (lower perplexity and higher commonsense reasoning performance, compared to fixed paramters/layer model at same FLOPs)
  • Tapering reduces throughput and increases memory consumption for the larger models
  • With off-the-shelf GPT models, authors show that the deeper layers only reinforce the residual direction, diminishing their contribution
Critical strengths
  • The paper is very well written and organized
  • The authors investigate a novel research question
  • Experiments are well structured and precisely answer the research question
  • All results consistently highlight the benefit of tapering with improvements in in-domain data and downstream tasks
Critical weaknesses
  • The paper does not investigate why deeper layers do not add value. Section 4 shows that deeper layers only reinforce the residual direction, but the 'why' is not answered. I completely understand if answering why is out of scope of the study. However, it might be helpful to make this clear in the manuscript.
  • I would like to see more discussion on how practitioners must tackle the reduction in throughput. For example, for one GPU inference, practitioners will need to set the batch size based on the first layer, which may lead to underutilization of the GPU during deeper layers' forward pass. For a multi-GPU setup, layers could be separated, and this issue can be partially resolved. However, the earliest layer will be the throughput bottleneck; it will play a part in setting the batch size, or it will decide the rate at which the next layer receives the requests. Please correct me if I am wrong.
  • The reasoning behind using older architectures (section 4: GPT-2, otherwise: llama-2) for the experiments is not clear
  • Because the main control variable (FFN intermediate size) will have a direct impact on MoE architectures, the paper may benefit from including MoE models in the experiments
Questions for authors
  • line-51: what is surface diversity?
  • line-375: given that one variable (layer index) in the correlation test is ordinal, Spearman rho might be more suitable? Please report p-values as well.
  • Please consider improving the readability of the plots. Currently, the numbers and the text on the plots are not readable.
5 comments

We thank the reviewer for their thoughtful feedback. Changes made in response are highlighted in light blue in the updated PDF (we also refer to them below).

No “why” for deeper layers adding less

We agree that Section 4 characterizes what happens (later MLP updates become increasingly aligned with the residual stream) rather than why. While we also believe a full answer to this is beyond the scope of this study, one candidate explanation comes from the “curse of depth” phenomenon (Sun et al., 2025). Modern language models use Pre-LN, because normalizing the residual stream itself (Post-LN) hampers gradient flow, which makes deep models difficult to train. In Pre-LN models, each layer’s output is added to an unnormalized residual stream, so the variance of the residual stream grows with depth. As a result, each layer’s update becomes small relative to the residual stream, and deeper layers increasingly behave like identity mappings. This account concerns the magnitude of layer updates, while our analysis measures their direction, so the two are complementary. Both point to the same practical conclusion: since later layers contribute less, it matters to allocate capacity across depth more carefully. We have added this discussion, including the scope of our analysis, to the end of Section 4 (highlighted in blue).

Practical guidance on the throughput cost

Thank you for raising this. We would first like to clarify that the throughput reduction in Table 3 is measured during training. As tapering keeps total FLOPs identical, this reduction does not come from additional computation; we expect most of it comes from the implementation. Our standard training implementation (e.g., fused kernels and their tuned configurations) is optimized for a stack of identical layers, and layers of different shapes break this assumption. A practical fix is to select kernel configurations separately for each layer width and to round widths to multiples of larger tile sizes (e.g., 64 or 128 instead of 16), so that every layer’s matrix multiplications run efficiently. As the reduction was small in our experiments (at most 6%), we did not optimize kernels for tapered models in this work. At larger scales, however, this cost matters more, and it can be addressed without changing the model. These fixes apply to both training and inference.

Regarding single-GPU inference, we would also like to clarify that the batch size is not set by the first layer. Tapering keeps total parameters and FLOPs fixed and does not change the attention layers, so the model weights and the KV cache, which dominate memory at inference, are unchanged. Moreover, since layers run one after another, the narrower later layers do not leave the GPU idle; they simply finish faster, and any efficiency cost from their smaller matrix multiplications can be reduced with the optimized kernels described above.

For multi-GPU settings, tensor parallelism (Shoeybi et al., 2019), widely used in both training and serving, splits each MLP’s intermediate dimension across GPUs, so every GPU processes every layer and no layer becomes a bottleneck. If layers are instead split across GPUs (pipeline parallelism), the reviewer is right that an even split would make the first stage the bottleneck. Because total compute is unchanged, this can be avoided in both training and inference by partitioning layers according to their FLOPs rather than their number, for example by assigning fewer of the wide early layers to the first stage.

We have added this practical guidance for both training and inference to Section 3.4 (highlighted in blue).

Why “older” architectures

Llama-2: only the tokenizer comes from Llama-2, as part of the standard recipe used in recent architecture studies (Titans, Gated DeltaNet). The models themselves are recent: Transformer++ (a Llama-style architecture with RoPE, SwiGLU, and RMSNorm), Gated Attention (Qiu et al., 2025), and Titans (Behrouz et al., 2024). We have made this clearer in Section 3.1.

GPT-2: we chose it for the analysis in Section 4 because it is a controlled family with publicly available pretrained checkpoints at four sizes (124M to 1.5B) that share the same architecture and training data. To show that the pattern is not specific to GPT-2, we extended this analysis to a modern open model family, Qwen2.5 (0.5B, 1.5B, 7B, 14B), and found the same trend: MLP outputs become progressively more aligned with the residual stream in the second half of the network (Appendix B, Figure 6, shown next to GPT-2 in Figure 5; Appendix B is highlighted in light purple, as it also addresses Reviewer PBFF). We also repeated the analysis for both families with 128× more tokens (262,144 instead of 2,048 WikiText-2 tokens), and the curves and correlations are essentially unchanged (Appendix B, Figures 7 and 8).

MoE

We agree that MoE models are a natural next step, since tapering acts on the FFN, which MoE replaces with experts. However, MoE introduces several additional per-layer axes (number of experts, expert width, and number of active experts) that interact with tapering, and a controlled study would require its own experimental design. We have added this as future work in the Discussion section.

“Surface diversity”

By surface diversity, we mean the differences in the token-mixing module across model families (e.g., softmax attention, gated attention, recurrent and state-space layers, or memory modules), while the block structure itself (a token-mixing module followed by an MLP, stacked uniformly across depth) is shared. We have rephrased this sentence in the Introduction as “despite their differences in token mixing.”

Correlation statistics

Thank you, we agree. We now report Spearman's rank correlation (rsr_s) between layer index and the mean cosine similarity, with two-sided p-values from a permutation test (200,000 permutations), and have replaced the Pearson correlations in Section 4 accordingly (Appendix B, Table 5). The table below uses 262,144 tokens; with the 2,048 tokens of the submission, the all-layer rsr_s values change by at most 0.03. As in Figure 4, the first and last layers are excluded.

ModelMLP, all layersMLP, second halfBlock, all layersBlock, second half
GPT-2 124M0.61 (p=0.066)1.00 (p=0.017)0.65 (p=0.049)1.00 (p=0.017)
GPT-2 355M0.69 (p<0.001)0.99 (p<0.001)0.42 (p=0.056)1.00 (p<0.001)
GPT-2 774M0.35 (p=0.042)0.94 (p<0.001)0.37 (p=0.030)0.97 (p<0.001)
GPT-2 1.5B0.51 (p<0.001)0.93 (p<0.001)0.46 (p=0.002)0.98 (p<0.001)
Qwen2.5 0.5B0.51 (p=0.017)0.63 (p=0.044)0.57 (p=0.006)0.63 (p=0.043)
Qwen2.5 1.5B0.57 (p=0.003)0.92 (p<0.001)0.58 (p=0.002)0.97 (p<0.001)
Qwen2.5 7B0.25 (p=0.22)0.80 (p=0.002)0.25 (p=0.22)0.86 (p<0.001)
Qwen2.5 14B0.27 (p=0.067)0.78 (p<0.001)0.32 (p=0.028)0.84 (p<0.001)

Over all layers, rsr_s is positive for every model and both quantities, and significant (p < 0.05) in 11 of 16 cases. The all-layer correlation understates the trend because the curves are not monotone: alignment first drops over the early layers and then rises steadily through the second half of the network (Figure 4), as described in Section 4. Restricted to the second half, rsr_s ranges from 0.63 to 1.00 with p < 0.05 for every model and both quantities. We report both in Appendix B (Table 5).

Figure readability

Thank you. The new figures in Appendix B use larger, full-width panels, and we will increase the size of the text and numbers in the remaining figures for the final version.

Dear Reviewer, as the discussion period closes tomorrow, we wanted to kindly follow up on our response. Based on your suggestions, we added new experiments and analyses that we believe have meaningfully strengthened the paper, and we would appreciate your feedback on whether they address your concerns.

Reviewer_PBFFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether a language model's MLP capacity should decrease with depth instead of remaining uniform. It proposes schedules that allocate larger intermediate widths to earlier layers and smaller widths to later layers while preserving total parameters and theoretical FLOPs. A sweep of 15 configurations on a 440M Transformer selects a cosine schedule ranging from 1.5 times to 0.5 times the baseline width, reducing validation perplexity from 16.28 to 14.44. That schedule is transferred without further tuning to Transformer, Gated Attention, and Titans models at 760M and 1.3B parameters. All six comparisons improve average benchmark accuracy and the two reported perplexities. A residual alignment analysis offers a possible explanation, while throughput and memory measurements expose practical costs of heterogeneous widths.

Critical strengths

The paper turns a broad observation about layer importance into a clean, controlled architectural experiment. Varying only MLP intermediate width avoids conflating capacity allocation with changes to attention heads, residual dimension, or model depth. The budget constraint preserves both parameter count and theoretical FLOPs, and the initial comparison tests whether early, middle, or late allocation is preferable. The schedule sweep then shows that the result depends on allocation shape: the best cosine configuration gives a substantial perplexity reduction at 440M, and all five tested cosine ranges improve over the uniform baseline. This is a consequential design result because it exposes an improvement available through redistributing existing capacity rather than increasing model size.

Critical weaknesses
  1. The robustness and practical value of the smaller transfer gains are not quantified. Several accuracy gains are only around 0.3 points, and the 1.3B Gated Attention perplexity changes from 16.03 to 15.92 on WikiText. Their magnitude relative to training variability is therefore unclear. At the same time, Table 3 reports approximately 6% lower throughput and 17% higher peak memory at 1.3B. Equal theoretical FLOPs consequently do not establish better performance at a fixed elapsed training time or memory budget. This uncertainty limits the strength of the practical recommendation.

  2. The explanation for the preferred direction remains indirect. The cosine similarity analysis uses only 2,048 tokens and GPT 2 checkpoints rather than the architectures in the transfer study. Alignment with a residual direction does not directly measure information content or the importance of an MLP contribution.

Questions for authors
  1. Evidence request: Was the reversed cosine profile, with identical widths and budget but increasing capacity with depth, evaluated under the same recipe? Does it support the preferred direction as clearly as the coarse allocation experiment?

  2. Clarification and evidence request: What exact model dimensions, initialization rules, hardware, batch settings, and checkpointing strategy produced Tables 2 and 3? When the measured throughput difference is included, does tapering still improve quality at a comparable elapsed training time?

5 comments

We thank the reviewer for their thoughtful feedback. Changes made in response are highlighted in light purple in the updated PDF (we also refer to them below).

Significance of the transfer gains.

Thank you for this suggestion. To quantify training variability, we trained 3 seeds of the uniform and tapered Transformer++ at 760M:

760M Transformer++Avg. accuracy ↑\uparrowWikiText ppl ↓\downarrow
Uniform52.31±0.2252.31 \pm 0.2222.03±0.1122.03 \pm 0.11
Tapered52.86±0.2452.86 \pm 0.2421.28±0.3021.28 \pm 0.30

(mean ±\pm standard deviation over 3 seeds). The gains (+0.55+0.55 accuracy, −0.75-0.75 perplexity) are larger than the seed-to-seed variability and statistically significant under a Welch tt-test (p≈0.04p \approx 0.04 for both accuracy and perplexity).

Furthermore, the gains reported in Table 2 are highly consistent across all comparisons: on each of the three aggregate metrics (average accuracy, WikiText perplexity, and LAMBADA perplexity), tapering improves in all six architecture/scale comparisons. If tapering had no effect, each comparison would be equally likely to go either way, and the probability of six out of six improvements on a metric would be (1/2)6=1/64≈0.016(1/2)^6 = 1/64 \approx 0.016 (sign test). We have added these results to Appendix C (highlighted in purple).

Practical cost and elapsed training time.

Thank you for raising this. At 440M, where the configuration was selected, training throughput is identical to the uniform baseline (Table 3), so the 1.841.84-point perplexity improvement in Table 1 is obtained at the same elapsed training time. The throughput reduction at larger scales is a byproduct of our implementation rather than of tapering itself, since tapering keeps total FLOPs identical. Our standard implementation is optimized for a stack of identical layers, and this can be addressed in two ways: selecting kernel configurations separately for each layer width, and rounding widths to multiples of larger tile sizes (e.g., 6464 or 128128 instead of 1616). As the reduction was small in our experiments (at most 6%6\%), we did not apply these optimizations in this work. Peak memory is controlled by the taper range: the narrower range 1.25→0.751.25 \rightarrow 0.75 reduces the widest layer while retaining 1.101.10 of the 1.841.84-point gain at 440M (Table 1). We have added this discussion to Section 3.4.

Layer-wise analysis.

Thank you for these suggestions. Following the reviewer's comments, we extended the analysis of Section 4 in the new Appendix B (highlighted in purple). First, we repeated it with 262,144262{,}144 tokens, 128×128\times more than the 2,0482{,}048 in the submission; the curves are essentially unchanged, and the all-layer correlations change by at most 0.030.03 (Figures 7 and 8). Second, we extended it beyond GPT-2 to Qwen2.5 (0.5B to 14B), a modern family with the same Llama-style block design as our Transformer++ (Pre-RMSNorm, SwiGLU, RoPE), and found the same trend: alignment rises through the second half of the network in every model (Figures 6 and 8).

Regarding importance, Section 4 is intended as an account consistent with our results, and direct importance measurements in prior work point in the same direction: early-exit and layer-removal studies show that later layers can be skipped or removed with little loss (Elbayad et al., 2020; Gromov et al., 2024; Men et al., 2025). The preferred direction itself is established directly by our controlled experiments: front-loading capacity helps and back-loading hurts, both in the coarse allocation (Figure 2) and in the reversed cosine profile below.

Reversed cosine profile.

Thank you for this suggestion. We trained the reversed cosine profile (0.5→1.50.5 \rightarrow 1.5: identical widths and budget, with capacity increasing with depth) for the 760M Transformer++ under the same recipe (Appendix C, highlighted in purple):

760M Transformer++Avg. accuracy ↑\uparrowWikiText ppl ↓\downarrow
Uniform52.31±0.2252.31 \pm 0.2222.03±0.1122.03 \pm 0.11
Cosine taper (1.5→0.51.5 \rightarrow 0.5)52.86±0.2452.86 \pm 0.2421.28±0.3021.28 \pm 0.30
Reversed cosine (0.5→1.50.5 \rightarrow 1.5)50.37±0.9450.37 \pm 0.9423.40±0.3823.40 \pm 0.38

(mean ±\pm standard deviation over 3 seeds). The reversed profile performs worse than both the tapered model and the uniform baseline, reducing average accuracy by 2.492.49 points and increasing perplexity by 2.122.12 relative to the tapered model. This confirms at a larger scale the direction found with the coarse allocation at 440M in Figure 2, where the wider-late configuration raised perplexity to 17.2917.29. Under the same widths and budget, front-loading capacity is the direction that helps.

Experimental details.

Thank you for this request. We have added the exact configuration of every model to Appendix D (highlighted in purple):

ModelLayersdmodeld_{\text{model}}HeadsHead dimTokens
440M241024812830B
760M2415361212850B
1.3B24204816128100B

Per-layer widths of the tapered models follow the cosine schedule (Equation 6) from 1.5 dff1.5\,d_{\text{ff}} to 0.5 dff0.5\,d_{\text{ff}}, rounded to multiples of 1616 as described in Section 2.2.

  • Initialization: all weights are drawn from N(0,1)\mathcal{N}(0, 1) and scaled by 1/d1/\sqrt{d}, identical for uniform and tapered models.
  • Hardware: H100 GPUs and TPUs.
  • Batch settings: a global batch of 0.50.5M tokens, with micro-batch sizes chosen to maximize memory utilization, identical for uniform and tapered models.

Dear Reviewer, since the discussion period ends tomorrow, we wanted to kindly check in. In our response, we included the additional experiments you suggested: results across three seeds, the reversed cosine profile, and an extension of the layer-wise analysis to Qwen2.5, with both GPT-2 and Qwen2.5 re-evaluated on 128× more tokens. We also added the full model configurations and training details. We hope these resolve your concerns and would appreciate hearing your thoughts.

Reviewer_SbDfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a relatively simple improvement of the standard transformer architecture -- progressively lowering the intermediate width of MLP modules. Tested on language models, applied to different architectures and model scales, this change consistently outperforms the standard practice of keeping the intermediate width constant.

Critical strengths
  • The paper is easy to follow and well-written. The authors applied their idea to three different transformer architectures (transformer++, gated attention, titans), and observed consistently positive results, which greatly supports the claims about their proposed improvements.
  • I appreciate the additional investigation of "layer-wise novelty". This observation motivates/explains the progressive lowering of width really well.
Critical weaknesses
  • The paper would be much stronger if it established clear scaling laws regarding the tapered architecture, now one can only guess how it would act at scale. This is especially important because even the current results suggest diminishing returns at larger scales.
Questions for authors
  • How were the pretraining hyperparameters determined? Is it possible that the observed performance is just a results of suboptimal training choices?
  • The "tapering" itself is well-motivated in the paper, but how do you explain the large performance differences between different tapering schedules? I found that very surprising.
3 comments

We thank the reviewer for their thoughtful feedback. Changes made in response are highlighted in light yellow in the updated PDF (we also refer to them below).

Scaling laws.

We agree that an extensive scaling-law study is a proper approach for predicting a model’s performance. At the same time, the generalizability of an approach, particularly in LLM architecture, is becoming an important consideration, since many variants of models are emerging, such as hybrid ones (e.g., attention combined with RNN-style components). Therefore, given our academic compute budget, we deliberately prioritized breadth across these variants to test whether the front-loading paradigm is architecture-agnostic. Our three backbones (softmax attention, gated attention, and neural long-term memory) cover important recent components in the field, and tapering improves average accuracy and both perplexities in all six architecture/scale comparisons. This provides practitioners with evidence that the approach transfers across settings. Nonetheless, we see a full scaling-law study tailored to each of these backbones as important future work, and we have stated this explicitly in the revised Limitations section (Section 6, highlighted in yellow).

Regarding diminishing returns, we agree that the gains at 1.3B are somewhat smaller than at 760M for some backbones. We would like to note that our configuration (a cosine schedule with 1.5 → 0.5 tapering range) for the large-scale runs (760M and 1.3B) was selected once based on a 440M Transformer++ model and transferred without re-tuning to other variants and scales. We do not rule out the possibility that the best taper range may shift with scale and variant (as we mention in Section 6 of the paper), and the gains reflect an untuned configuration rather than the best achievable taper at that size and variant.

Hyperparameters.

We adopted the standard recipe widely used in studies of language-model architectures (Titans (Behrouz et al., 2024) and Gated DeltaNet (Yang et al., ICLR 2025)). Here is the recipe (also in the paper): FineWeb-Edu dataset, Llama-2 tokenizer (with 32K vocabulary), 4K training context length, 0.5M-token batches, AdamW with a peak learning rate of 4e-4, cosine decay with a 1B-token warmup, and weight decay of 0.1.

Regarding potential “suboptimal training choices”: the above recipe was developed for uniform-width models in the literature, and we applied it unchanged to our tapered variants without any additional tuning. To the extent the recipe is tuned at all, it is tuned for uniform-width models, so any tuning bias works against tapering rather than in its favor. Please also note that the tapered models use a single global learning rate for layers whose widths differ by up to 3×, which is likely not optimal. In the revised PDF, we now stated explicitly that no hyperparameters were tuned for the tapered models, and we also added the warmup schedule, which was omitted from Section 3.1 (highlighted in yellow).

Differences between schedules.

In Table 1 linear schedule performs better than sigmoid at every range, although that gap is small; however, cosine is the best schedule at every range. Since all three schedules share the same endpoints and are symmetric around the middle of the network, they differ only in “how quickly width changes across depth”. Linear starts shrinking from the first layer at a constant rate, which reduces the capacity of early layers very soon. Sigmoid, on the other hand, preserves the capacity of early layers but it’s close to a single step between two widths in the middle of the network (meaning it has a sharp transition between capacities). This sharp transition resembles the step-wise allocation studied in Figure 2, and the two perform similarly (16.12 vs. 15.96 at 1.5 → 0.5). These observations therefore suggest that a suitable schedule should both “preserve the capacity of early layers” and “avoid sudden changes in width across layers”. Cosine satisfies both. We have added this clarification to Section 3.2 of the revised paper (highlighted in yellow).

Dear Reviewer, with the discussion period ending soon, we wanted to kindly follow up on our response regarding hyperparameters, schedule differences, and scaling. If any concerns remain, we would be happy to address them.

AuthorsOpenReview ↗

Follow-up: Summary of Revisions

Dear Reviewer, as the discussion period closes soon, we have posted a brief summary of all revisions above. We would be grateful if you could take a look. Thank you again for your time and feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear ACs and Reviewers,

We thank all reviewers for their time and constructive feedback. Reviewer K1o8 has indicated that all concerns were addressed and raised their score to "clear acceptance". For the reviewers who have not yet responded, we summarize the main additions (all highlighted in the revised PDF):

Reviewer SbDf:

  • Training recipe: Stated explicitly that no hyperparameters were tuned for tapered models (the recipe was developed for uniform models, so any bias favors the baseline); added the warmup schedule. (Sec. 3.1)
  • Schedule differences: Explained why cosine outperforms linear and sigmoid: it preserves early-layer capacity while avoiding sharp width transitions. (Sec. 3.2)
  • Scope: Scaling laws and scale-dependence of the best taper range are discussed as future work. (Sec. 6)

Reviewer PBFF:

  • Statistical robustness: 3-seed runs at 760M show tapering gains (+0.55 acc, −0.75 ppl) exceed seed variability (Welch t-test p ≈ 0.04); tapering improves average accuracy, WikiText ppl, and LAMBADA ppl in all 6 architecture/scale comparisons. (App. C)
  • Reversed cosine profile: Same widths and budget, but increasing with depth: worse than both the taper (−2.49 acc, +2.12 ppl) and the uniform baseline, confirming the preferred direction at 760M. (App. C)
  • Layer-wise analysis: Extended to Qwen2.5 (0.5B–14B) and 128× more tokens; curves essentially unchanged. (App. B)
  • Practical cost: Throughput is identical to the baseline at 440M; we added guidance on per-width kernel tuning and controlling memory via the taper range. (Sec. 3.4)
  • Experimental details: Full model configurations, initialization, hardware, and batch settings added. (App. D)

Reviewer K1o8:

  • Layer-wise analysis: Pearson replaced with Spearman correlations and permutation-test p-values, significant in the second half of the network for every model and both quantities; added a discussion of why later layers may add less (curse of depth in Pre-LN models). (Sec. 4, App. B)
  • Practical guidance: Added guidance for inference and tensor/pipeline parallelism. (Sec. 3.4)
  • Clarifications: Only the tokenizer comes from Llama 2, and the architectures are recent (Sec. 3.1). We clarified "surface diversity" (Sec. 1) and enlarged the new figures; the remaining figures will be enlarged for the final version.
  • Scope: MoE is discussed as future work. (Sec. 7)

We sincerely appreciate everyone's time and are happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.