I see two major weaknesses which needs to be addressed during the rebuttal period.
First, the residual norm growth in pre-LN architecture is a known phenomenon, and their exact mitigation is proposed in MixLN paper [1]. In particular, residual norm can be controlled by having post-LN in first few layers (e.g., 25% of layers), and rest as pre-LN. Authors must have discussed and further compare the pros and cons of simplex gating with MixLN approach.
Second, to test whether the residual update asymmetry is a fundamental property of Pre-Ln architecture or it emerges due to AdamW optimizer, authors needs to pre-train, at least one smaller LLaMA model (152M), with Muon/NorMuon optimizer. In fact, Dion has nice property to constrain the update rank (1/2, 1/4, 1/8, 1/16) and allow a trade-off between Muon and AdamW like optimization behavior.
This is particular important because earlier observations, prior to the advent of Muon optimizer, such as attention-induced rank collapse, were understood as fundamental property of transformer architecture. However, later studies, such as [2] showed it as specific to AdamW.
[1] Li et al., Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN, ICLR 2025
[2] Jha et al., NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks, ICLR 2026
Reminder: Author–Reviewer Discussion Deadline Approaching
Dear Reviewers and Authors,
Thank you to the reviewers for your time, effort, and dedication in reviewing this paper. This AC also appreciates the authors' detailed rebuttal, which addresses the reviewers' comments and includes additional clarifications and experimental results.
To the reviewers: As the author–reviewer discussion deadline approaches, please carefully review the authors' responses and assess whether they adequately address your concerns. If any concerns have not been fully addressed, please let the authors know what further clarification or evidence would be helpful.
Thank you again for your valuable contributions and constructive discussion.
Best,
Your AC