- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how feed-forward network (FFN) parameters should be allocated across Transformer layers. The authors propose a Sandwich-style architecture in which middle layers share FFN parameters, while early and late layers retain independent FFNs. The saved parameters are used to widen the FFNs, keep the total parameter count approximately fixed. By training 147 configurations from scratch at the 570M scale, the authors find that most configurations, especially those retaining more independent FFNs in later layers, improve downstream task loss over a standard Transformer. The authors also show the best configuration improves average downstream accuracy at the 2B scale, although it requires substantially more computation.
Critical strengths
- The paper studies a simple and clear research question: which layers of LLMs need independent FFNs? The proposed method and experiments are clearly presented and easy to reproduce.
Critical weaknesses
- The authors motivate their method using prior findings about layer-wise representations, but I find this motivation rather high-level and insufficient to justify the specific design. There is also limited analysis of why later FFNs may need more independent parameters. The authors propose two hypotheses in Section 6 but do not provide experiments to verify them, which I consider a major weakness.
- The paper does not show that the proposed method outperforms the baseline under matched FLOPs. I understand that pretraining models from scratch is computationally expensive, but given the relatively small performance improvements and the use of only a single training seed, it is difficult to assess how robust the gains are. The authors could consider approaching this research question from a slightly different angle, with more emphasis on understanding the observed behaviour.
Questions for authors
- The motivation in Section 3.2 relies mainly on prior observations about layer-wise representations and redundancy. Can the authors provide more direct evidence or additional experiments supporting the proposed sharing strategy, particularly why sharing FFNs across middle layers should be effective?
- Can the authors provide additional analysis to test either of the two hypotheses proposed in Section 6 for why late-heavy configurations perform better?
Reminder: Author–Reviewer Discussion Deadline Approaching
Dear Reviewers and Authors,
Thank you to the reviewers for your time, effort, and dedication in reviewing this paper. This AC also appreciates the authors' detailed rebuttal, which addresses the reviewers' comments and includes additional clarifications and experimental results.
To the reviewers: As the author–reviewer discussion deadline approaches, please carefully review the authors' responses and assess whether they adequately address your concerns, if you haven't already. If any concerns have not been fully addressed, please let the authors know what further clarification or evidence would be helpful.
Thank you again for your valuable contributions and constructive discussion.
Best,
Your AC