acceptodds
Under review as a conference paper at ICLR 2027

Variable-Width Transformers

34 views

Abstract

Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across layers, allocating a fixed parameter and computation budget despite their potentially different computational roles. We empirically investigate nonuniform capacity allocation by proposing a ×-shaped > <former architecture. It maintains wider early and late layers while narrowing the middle layers, using a parameter-free residual resizing mechanism. Across decoder-only language models ranging from 200M to 2B parameters (dense) and 3B parameters (MoE), our > <former consistently outperforms parameter-matched uniform baselines on language modeling loss. By reducing the average layer width, this architecture also requires fewer overall FLOPs (22% reduction under fitted loss-matched scaling curves) and smaller KV cache memory and I/O cost (15% reduction). Consequently, dense > <formers reach the uniform baseline’s final smoothed training loss with 15.6–23.1% fewer profiled GPU-hours, and generation throughput improves by 8.8% in one profiled setting. In analysis, we find qualitatively different residual-stream representations. Overall, nonuniform width allocation can result in more resource-optimal scaling of language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
transformer architectures, large language models, efficient pretraining

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_drDXOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper explores whether Transformer layers need to be equally wide. It proposes ><former, with wider early and late layers and narrower middle layers, while carrying inactive residual dimensions forward. Under matched parameter budgets, it reduces language modeling loss across several model sizes with fewer FLOPs and smaller KV caches. Downstream results are generally better, and the experiments also show practical efficiency gains in dense models.

Critical strengths
  • The paper tackles a meaningful architecture question: how to distribute capacity across Transformer layers. The proposed design is simple, and its gains under matched parameter budgets suggest that uniform width is worth reconsidering.
  • The experiments are extensive and well connected to the main claims. Comparisons across model sizes, width configurations, and residual resizing methods support the design, while held-out and downstream evaluations show that the benefits extend beyond training loss.
Critical weaknesses

The hyperparameter search settings for the expansion method ablation in Table 2 are not fully specified. Although the authors mention that multiple configurations were explored for each method, it is unclear whether the same search space and tuning budget were used, making it difficult to assess the fairness of the comparison.

Questions for authors

For the expansion method ablation in Table 2, could the authors clarify whether the same hyperparameter search space and search budget were used across the different methods? In particular, were the bottleneck locations and widths searched over the same candidate configurations?

2 comments

Thank you for highlighting the significance of our method and its simplicity. To answer your question about the hyperparamter settings for Table 2: each expansion method sweeps over 3 bottleneck layer indices. They use the same bottleneck width ratio rd=0.3r_d=0.3, training budget, and other training settings. Across each bottleneck index, we observe the same carry-forward < zero-padding < learned projection loss comparison, demonstrating the consistency of this pattern.

Bottleneck blockCarry-forwardZero-paddingLearned projection
12 of 24 (50%)3.0963.1483.157
18 of 24 (75%, paper default)3.0993.1233.130
21 of 24 (87.5%)3.1103.1293.132

Thank you for the clarification and the additional experimental results. The matched comparisons across different bottleneck locations address my concern about the fairness of the ablation. I will maintain my original rating.

Reviewer_NaJyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper questions the convention of constant width across transformer layers and proposes the > <former, a variable-width transformer with wide early/late layers and a narrow middle. Across parameter-matched conventional transformers from 200M to 2B, the authors report lower loss, fewer training FLOPs, reduced KV cache, and fewer training GPU-hours.

Critical strengths
  1. The question is well-motivated, since nearly all transformer LMs keep width constant across depth, and the parameter-free carry-forward construction is simple and outperforms zero-padding and learned projections in the ablation.
  2. There is a comprehensive analysis of the method, covering a wide range of model scales, both dense and MoE. The analysis includes downstream task performance, scaling laws, and representation analysis.
  3. Their efficiency claims are strongly supported, with an analysis showing that variable-width architectures provably reduce FLOP count and KV cache size when parameter-matched against a constant-width baseline.
Critical weaknesses
  1. The paper does not isolate the effect of variable width from the effect of the skip connections. The residual connections used in this paper are effectively U-Net skips, which are already a known technique for improving training efficiency, used for example in the modded-nanogpt speedrun [1] and OpenAI Parameter Golf challenge [2]. A uniform-width transformer with U-Net skips should be added as a baseline, ideally with the full analysis if possible (all scales, scaling fits, downstream results, and GPU-hours).
  2. It is hard to interpret the FLOP savings with the GPU-hour savings. > <formers only save about 3% of FLOPs, but they take around 20% fewer GPU-hours to reach the baseline loss. It is unclear how much of the GPU-hour savings comes from hardware utilization and how much comes from fewer training steps. Intuitively, I would expect lower hardware utilization since this paper narrows the middle layers, but the low GPU-hours suggests this is not the case. Per-step wall-clock time and training throughput (t/s or MFU) is needed to clarify this.

[1] modded-nanogpt, https://github.com/KellerJordan/modded-nanogpt.

[2] Parameter Golf, https://github.com/openai/parameter-golf

Questions for authors
  1. Could the authors compare the > <former against a uniform-width transformer with U-Net skip connections, at least at 500M? This would separate the effect of the bottleneck from the effect of carrying early-layer features to late layers.
  2. What are the per-step wall-clock time and training throughput for each architecture, and how were the profiled GPU-hours in Table 1 computed?
  3. What is the seed-to-seed variance of loss and average downstream accuracy at 500M? The reported improvements are quite small so it is important to see whether it is from noise or true gains.
6 comments

We really appreciate that you recognize the strengths of our results! We are also grateful for your feedback: we have correspondingly added new results and discussions in our draft, which strengthen the paper. Specifically:

1. Width variation and skip connections. Our current results in Table 2 already show the effect of varying width vs. skip connections. At 500M, a > <former variant that uses zeros to pad new dimensions (i.e., no skip connection) improves loss from 3.138 to 3.123; adding carry-forward skip connection further improves it to 3.099. This shows that the two mechanisms stack.

As you suggested, we also tested adding U-Net-style skip connections to a baseline constant-width model. Whereever there is a skip connection in our corresponding 500M > <former, we connect the same source and destination layers in the constant-width baseline, rescaling the skipped channel range in proportion to the baseline’s residual width. This lowered loss from 3.135 to 3.110, but still higher than 3.099 for the > <former. This result once again shows that the two mechanisms are complementary. Furthermore, as the skip connection variant keeps the constant-width shape, it uses more FLOPs and KV cache size than a > <former.

2. Training throughput and GPU-hours. At 1B (16 H100s), > <former takes 2.9% more time per step than the baseline (2.46 vs. 2.39 s/step, or 0.85M vs. 0.88M tokens/s) but needs about 25% fewer steps, reaching the same loss in 23.1% less profiled training time (194 vs. 253 GPU-hours). At 2B (64 H100s), > <former takes 5.5% more time per step than the baseline (2.81 vs. 2.66 s/step, or 1.49M vs. 1.57M tokens/s) but needs 20% fewer steps (20,000 vs. 25,000), reaching the same loss in 15.6% less profiled training time (999 vs. 1,184 GPU-hours). We have added these details to the paper.

3. Variation across training seeds. We repeated the 500M comparison with three matched training seeds. > <former reached a lower final training loss than the baseline at all three seeds. This small variation across pretraining seeds is consistent with prior studies: Gupta et al. [1] report a validation-loss SD of 0.001 across five 405M runs, and Bergsma et al. [2] report SDs below 0.003 at each of six 111M training configurations. For downstream accuracy, DataDecide [3]'s three 530M DCLM-Baseline runs have an across-seed SD of 0.115 percentage points in ten-task mean accuracy. Su et al. [4] report SDs of 0.2 and 0.5 points in ten-task mean zero-shot accuracy for their 1.3B and 124M baselines.

SeedBaseline> <formerDifference (> <former − baseline)
423.13543.1019−0.0335
433.13273.1028−0.0298
443.13493.0995−0.0354
Mean ± sample SD3.1343 ± 0.00153.1014 ± 0.0017−0.0329 ± 0.0028

References

[1] Gupta et al. On Quantizing the State of the Muon Optimizer. ICLR SPOT Workshop, 2026.

[2] Bergsma et al. Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training. NeurIPS, 2025.

[3] Magnusson et al. DataDecide: How to Predict Best Pretraining Data with Small Experiments. ICML, 2025.

[4] Su et al. Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization. arXiv:2608.11746, 2026.

Thank you again for your helpful feedback! We've added the matched-skip comparison and three-seed results, and clarified the training throughput and GPU-hour savings in our response and the revised paper. If you have a chance to take another look, we would appreciate hearing whether these address your concerns. We'd be happy to clarify any remaining questions.

Thank you for your detailed response, I believe the contributions of varying transformer width are now clear. I have two follow-ups:

  1. I noticed that the overhead per step increases from 1B to 2B, what about for the 200M and 500M sizes? I am wondering if there is a trend that points to larger overheads for larger model sizes.
  2. Since you have trained 500M models at three different seeds, can you report the downstream task accuracy of each seed and find the across-seed SD? This shouldn't take much compute since your downstream tasks are either multiple choice or word prediction. I think this would provide stronger evidence compared to citing the SD of other papers, since the works you cited use different collections of downstream benchmarks.

I am willing to raise my score after these two points have been addressed.

Reviewer_mmcBOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces ><former, which reduces the channel width of intermediate layers in a standard transformer. The experimental setup appears sound and is representative enough to be informative to frontier LLM development. Other variants such as diamond profiles, narrowing or widening width are also swept, but the hourglass structure generally works best. Both the main experiments as well as the subsequent analysis provide lots of interesting insights.

Critical strengths
  • Variable width transformers are an interesting idea with the potential to inspire a lot of follow up investigations and opens up a new design space for future iterations of LLMs. I believe lots of people would be interested to pass by such thought provoking work and discuss it in person at ICLR.
  • Experiments are described in detail + also reports on variations that did not work out. The results are generally very convincing and analyze a lot of different aspects of the proposed architectural change.
Critical weaknesses

None

Questions for authors

The paper was very clear, no substantial questions to ask that would change my assessment. Still, I would be interested on the authors take on whether variable width could be applied to hybrid models. Could one operate the full attention layers with lower channel width to reduce attention cost, while operating SWA or linear attention blocks on the full residual stream - at least in the middle layers?

2 comments

Thank you for highlighting the value of the thoroughness of our experiments and results. We believe that reporting width schedules that performed worse also provides empirical evidence that would guide future work.

We believe that your suggestion to combine variable width with hybrid attention is very promising. Linear attention summarizes past tokens in a single recurrent state, so it is reasonable to keep those layers wide so that the reucrrent state can be more expressive. Narrower full-attention layers in the middle could then reduce the cost of attending over long contexts. We would be interested to see how this allocation works in practice as future work.

Thank you for your perspective. No further questions from my side, I will maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.