- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper revisits the narrow-wide-narrow FFN convention in dense Transformer language models and proposes Hourglass Transformers, which flip the hidden size from narrow-wide-narrow to wide-narrow-wide. Across 906M, 3B, and 8B models, the authors report comparable validation and downstream quality, 2-4% fewer training FLOPs, up to 1.93x faster decoding, and 50% smaller KV cache at 64k context.
Critical strengths
- The question is well-motivated, since the FFN shape has gone largely unchanged in dense LLMs.
- The evaluation methodology spans both perplexity measurements as well as downstream task performance, long-context extension, and inference measurements, providing a comprehensive analysis of the hourglass-style architecture.
Critical weaknesses
- Many components of this work are not new, and a substantial body of related work is missing:
- Stacking several FFN subblocks per attention block has already been done in MobileBERT [1].
- Numerous pruning works [2] have found that large portions of attention layers exhibit high similarity with each other and can be removed without degrading performance, which supports the above point of having multiple FFN blocks per attention block.
- Hourglass attention amounts to decoupling d_model from the attention width, which is already very common in modern LLMs like Qwen 3, Gemma 4, etc.
- The baselines do not isolate the effect of the hourglass architecture. Notably, the layer count changes between the hourglass and conventional transformers, which largely explains the decode speedups and kv cache memory reductions. The effect the hourglass architecture specifically has on inference efficiency is unclear. The more informative question would be to compare the hourglass and conventional transformer architectures at a fixed layer count, which would account for these differences and provide a better understanding of the differences between the architectures.
- The 8.7% training-compute claim is weak, since it comes from fitting a 2-parameter scaling law on three data points per architecture. With so few data points, it is unclear whether the 8.7% is an accurate representation of the gains, or if it is just noise.
- The narrower bottleneck matmuls and K sequential sub-blocks may lead to lower hardware utilization. This figure was not reported in the paper, and it may negate the small FLOP advantage seen in this architecture.
- The benchmark gains are small and likely within margin of error, so error margins should be reported. Also, many benchmarks are near-chance, where 1-2% differences are not very meaningful.
[1] MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices, https://arxiv.org/abs/2004.02984.
[2] What Matters in Transformers? Not All Attention is Needed, https://arxiv.org/abs/2406.15786.
Questions for authors
- Could the authors compare Hourglass and conventional models at the same layer count, at least at 906M? This would separate the effect of the hourglass FFN from the effect of simply using shallower, wider networks.
- What is the seed-to-seed variance of validation PPL and average downstream accuracy at 906M?
- What are the measured training throughput (tokens/sec or MFU) and activation memory for each architecture?