acceptodds
Under review as a conference paper at ICLR 2027

Revisiting the Shape Convention of Transformer Language Models

Abstract

The architectural shape of dense Transformers has remained remarkably stable: narrow-wide-narrow feed-forward networks (FFNs) consume most non-embedding parameters. Motivated by theoretical and empirical evidences that residual wide-narrow-wide (hourglass) MLPs remain expressive despite bottlenecks, we revisit whether this architectural convention is necessary for dense language models. We study Hourglass Transformers, which replace the conventional FFN with residual stacks of hourglass sub-MLPs and use hourglass attention to decouple residual-stream width from attention width. This exposes a practical depth-width trade-off: compressing the FFN intermediate dimension allows wider hidden states and fewer layers at matched parameter budgets. Across model scales from 113M to 8B parameters, Hourglass Transformers achieve language-modeling and downstream performance comparable to conventional Transformers, while improving training compute efficiency by at matched average downstream accuracy across the 906M, 3B, and 8B scales. After long-context extension, the 8B Hourglass model also outperforms its matched conventional baseline across 4k–64k context lengths. At 64k context, the reduced attention layer count lowers both computation and KV-cache requirements, yielding up to faster token decoding and lower KV-cache memory at the 1B scale. These results identify hourglass structures as a practical architecture-efficiency alternative for compute- and latency-conscious Transformer design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformer architecture, hourglass MLP, feed-forward networks, depth-width trade-off

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FvRmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper revisits the narrow-wide-narrow FFN convention in dense Transformer language models and proposes Hourglass Transformers, which flip the hidden size from narrow-wide-narrow to wide-narrow-wide. Across 906M, 3B, and 8B models, the authors report comparable validation and downstream quality, 2-4% fewer training FLOPs, up to 1.93x faster decoding, and 50% smaller KV cache at 64k context.

Critical strengths
  1. The question is well-motivated, since the FFN shape has gone largely unchanged in dense LLMs.
  2. The evaluation methodology spans both perplexity measurements as well as downstream task performance, long-context extension, and inference measurements, providing a comprehensive analysis of the hourglass-style architecture.
Critical weaknesses
  1. Many components of this work are not new, and a substantial body of related work is missing:
  • Stacking several FFN subblocks per attention block has already been done in MobileBERT [1].
  • Numerous pruning works [2] have found that large portions of attention layers exhibit high similarity with each other and can be removed without degrading performance, which supports the above point of having multiple FFN blocks per attention block.
  • Hourglass attention amounts to decoupling d_model from the attention width, which is already very common in modern LLMs like Qwen 3, Gemma 4, etc.
  1. The baselines do not isolate the effect of the hourglass architecture. Notably, the layer count changes between the hourglass and conventional transformers, which largely explains the decode speedups and kv cache memory reductions. The effect the hourglass architecture specifically has on inference efficiency is unclear. The more informative question would be to compare the hourglass and conventional transformer architectures at a fixed layer count, which would account for these differences and provide a better understanding of the differences between the architectures.
  2. The 8.7% training-compute claim is weak, since it comes from fitting a 2-parameter scaling law on three data points per architecture. With so few data points, it is unclear whether the 8.7% is an accurate representation of the gains, or if it is just noise.
  3. The narrower bottleneck matmuls and K sequential sub-blocks may lead to lower hardware utilization. This figure was not reported in the paper, and it may negate the small FLOP advantage seen in this architecture.
  4. The benchmark gains are small and likely within margin of error, so error margins should be reported. Also, many benchmarks are near-chance, where 1-2% differences are not very meaningful.

[1] MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices, https://arxiv.org/abs/2004.02984.

[2] What Matters in Transformers? Not All Attention is Needed, https://arxiv.org/abs/2406.15786.

Questions for authors
  1. Could the authors compare Hourglass and conventional models at the same layer count, at least at 906M? This would separate the effect of the hourglass FFN from the effect of simply using shallower, wider networks.
  2. What is the seed-to-seed variance of validation PPL and average downstream accuracy at 906M?
  3. What are the measured training throughput (tokens/sec or MFU) and activation memory for each architecture?
Reviewer_KnQ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper challenges the narrow–wide–narrow FFN convention in dense Transformers by proposing Hourglass Transformers. These replace each FFN with K stacked residual wide–narrow–wide SwiGLU sub-MLPs and use “hourglass attention,” which projects Q/K/V from a wider residual stream down to a baseline-matched d_attn. Shrinking the FFN bottleneck frees parameters, which are reallocated to a wider d_model and fewer layers. The authors validate this with a controlled 113M FFN-shape study, a frozen-LM-head study at 403M, a width/depth sweep at 906M, and parameter-matched comparisons at 906M, 3B, and 8B. They report comparable validation PPL and downstream accuracy, 2–4% fewer training FLOPs per pair, better Proof-pile PPL and RULER scores after YaRN extension at 8B, and 64k-context decode speedups of 1.93×/1.52×/1.43×. The KV-cache reduction equals 1 − L_hg/L_conv and holds across MHA, GQA, and MLA.

Critical strengths
  1. Practical inference benefit with a clean explanation. Because d_attn is matched to the baseline, KV memory and long-context decode cost scale with L. The paper derives this analytically (Eq. 7–9) and confirms it empirically across MHA, GQA, and MLA. Fewer layers at matched quality give a 31–50% KV-cache reduction and up to ~1.9× faster decode at long context, which matters for deployment.

  2. Evidence that the FFN shape convention is not essential, at up to 8B scale. The controlled 113M experiment isolates FFN shape and shows it is at least competitive. The 906M–8B results show quality parity under a substantially different width/depth allocation. This challenges a long-standing default and opens the design space.

Critical weaknesses
  1. The quality and compute-efficiency claims are not statistically established. The headline “8.7% less training compute” comes from a log-linear fit over only three points per architecture, while per-pair FLOP savings are 2–4%.

  2. The large-scale results do not attribute gains to the hourglass FFN. At 906M–8B, FFN shape, residual width, layer count, and attention width all change together. The KV and latency gains follow purely from the smaller L, which any shallower model would also get.

Questions for authors
  1. What are the measured training throughput and prefill latency compared with the baselines?
  2. Can you add a parameter-matched conventional baseline with L = 18 at 3B and L = 22 at 8B? This would separate the effect of the hourglass FFN from the effect of simply using fewer layers.
  3. The K-ablation is not parameter-matched (67M → 175M). Is there a parameter-matched version?
Reviewer_Vt4tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates “hourglass-shaped” FFNNs in transformers. Specifically, they explore a setting where the FFNNs in the transformer have (i) multiple layers (connected with their own residual connection) and (ii) reduce the dimension rather than expanding it as in traditional FFNNs (which gives them the hourglass-like shape). The paper then conducts an empirical investigation of this architectural change at 100M-8B scales, finding that they can lead to 8.7% more FLOP efficiency than regular FFNNs, as well as allowing for a wider residual stream in an (roughly) iso-parameter setting. They also conduct several ablations to study other aspects of their approach (e.g., compatibility with GQA/MLA).

Critical strengths

The empirical study is fairly comprehensive. They conduct a thorough empirical study and include several ablations for different parts of their model. Furthermore, save perhaps for minor exceptions, the authors seem to have invested reasonable effort to adopt a reasonable baseline architecture and training recipe (App. 1). They include experiments at moderate scales (8B), which is valuable to understand how this method could scale to even larger settings.

Some promising results are shared. There are several promising results, such as improving training efficiency relative to regular transformers (via a scaling law argument), showing somewhat comparable PPLs (e.g., <1% worse at most) at slightly reduced training FLOPs, and positive long-context results.

Critical weaknesses

Some results are insufficiently supported. Some of the headline results are not sufficiently supported by the results. For example, the argument that Hourglass transformers can lead to higher decode efficiency, as far as I understand, seems to be based on reducing the number of layers in the overall transformer, but it is not entirely clear that hourglass FFNNs are critical to this finding: for example, it is not clear if this could have been achieved by a larger residual stream, or a depth-matched conventional baseline (perhaps with wider, but conventional, FFNNs?). Furthermore, another orthogonal approach for decode speed-ups is using GQA, but the authors do not report any results on GQA models that would lead to a corresponding throughput improvement. It also seems like the increased size of the residual stream is also possibly a substantial part of why this method is effective. Given this, it would be worthwhile to add, e.g., mHC as a baseline.

Some results are mixed. While there are some positive results, others seem more mixed. For example 4.2’s results don’t suggest a clear trend in the hourglass shape, and based on Table 2, it is not clear if the 8.7% training efficiency improvement would hold if we target PPL rather than average downstream accuracy as shown in Figure 4.

Methodological contribution is minor. While the results are interesting (and could potentially be strong if some of the above points are resolved), the method itself is still a fairly small change on top of conventional transformers.

Questions for authors

What is the training efficiency improvement (i.e., 8.7% in Figure 4) if we computed it against loss instead of average downstream accuracy?

Is there a reason why you chose to do 400 steps of YaRN long-context extension? How many tokens does that correspond to? Glancing at the RULER numbers, they seem a bit low—are these numbers in line with published figures at 8B scale?

Can you clarify the setup for the frozen LM head experiments? What exactly is frozen, and what is being trained in those experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.