- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper explores whether Transformer layers need to be equally wide. It proposes ><former, with wider early and late layers and narrower middle layers, while carrying inactive residual dimensions forward. Under matched parameter budgets, it reduces language modeling loss across several model sizes with fewer FLOPs and smaller KV caches. Downstream results are generally better, and the experiments also show practical efficiency gains in dense models.
Critical strengths
- The paper tackles a meaningful architecture question: how to distribute capacity across Transformer layers. The proposed design is simple, and its gains under matched parameter budgets suggest that uniform width is worth reconsidering.
- The experiments are extensive and well connected to the main claims. Comparisons across model sizes, width configurations, and residual resizing methods support the design, while held-out and downstream evaluations show that the benefits extend beyond training loss.
Critical weaknesses
The hyperparameter search settings for the expansion method ablation in Table 2 are not fully specified. Although the authors mention that multiple configurations were explored for each method, it is unclear whether the same search space and tuning budget were used, making it difficult to assess the fairness of the comparison.
Questions for authors
For the expansion method ablation in Table 2, could the authors clarify whether the same hyperparameter search space and search budget were used across the different methods? In particular, were the bottleneck locations and widths searched over the same candidate configurations?
2 comments
Thank you for the clarification and the additional experimental results. The matched comparisons across different bottleneck locations address my concern about the fairness of the ablation. I will maintain my original rating.
Thank you for highlighting the significance of our method and its simplicity. To answer your question about the hyperparamter settings for Table 2: each expansion method sweeps over 3 bottleneck layer indices. They use the same bottleneck width ratio rd=0.3, training budget, and other training settings. Across each bottleneck index, we observe the same carry-forward < zero-padding < learned projection loss comparison, demonstrating the consistency of this pattern.