- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes FlashNorm. Its main idea is to reorganize RMSNorm followed by a linear layer so that and can be computed in parallel, thereby accelerating computation. The method is also extended to LayerNorm, DyT, and certain attention operations. The paper discusses limitations involving bias and epsilon, and presents experiments to measure the acceleration.
Critical strengths
- The method is studied in the context of LLMs, which demonstrates its practical relevance.
- The paper clearly and honestly discusses limitations of the method, including the treatment of bias and epsilon in RMSNorm.
Critical weaknesses
The approach in Section 3.5 appears extremely similar to the prior work in [1] and [2], but neither paper is discussed.
[1] Yuxin Guo, Yihao Yue, Yunhao Ni, Yizhou Ruan, Jie Luo, Wenjun Wu, and Lei Huang. Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm. arXiv preprint arXiv:2605.14521, 2026.
[2] Zixuan Jiang, Jiaqi Gu, Hanqing Zhu, and David Pan. Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers. Advances in Neural Information Processing Systems 36, 2023.
Questions for authors
- Section 3.5 does not cite [1] or [2]. Could the authors discuss how Section 3.5 relates to these works and clarify whether it is intended as a contribution of this paper?
- Is the equality in line 212 correct? Should scale invariance instead be expressed as ?
- Could the authors explain what , , and denote in Figure 1? Their meanings are unclear to me from the current presentation. Symbols such as should also be defined when they first appear.
[1] Yuxin Guo, Yihao Yue, Yunhao Ni, Yizhou Ruan, Jie Luo, Wenjun Wu, and Lei Huang. Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm. arXiv preprint arXiv:2605.14521, 2026.
[2] Zixuan Jiang, Jiaqi Gu, Hanqing Zhu, and David Pan. Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers. Advances in Neural Information Processing Systems 36, 2023.
4 comments
Q1: Addressed. I think the authors can add the references and adjust the descriptions in sec 3.5.
Q2: Addressed. I think the original writing may leads confusions.
Q3: How about n,1, k? I have not read about such figures before so I do not understand it. I may guess they denote the operation , , , but I am not sure. Could the authors provide more detailed description about it?
Thank you for your prompt response, and for confirming that Q1 and Q2 are addressed. We will add the references, adjust the descriptions in Section 3.5, and clarify the scale-invariance statement.
Q3 (Figure 1). We apologize that the figure was not self-explanatory. The numbers , , and next to the slashes on the wires do not denote operations. They give the width of each wire, i.e. the number of elements it carries (the same convention as bus widths in hardware block diagrams):
- A wire labeled carries a vector with elements: the input , the normalized vector, and the weights .
- A wire labeled carries a vector with elements: the output of the linear layer.
- A wire labeled carries a single scalar: the box "1/RMS" takes the elements of and produces one number per token.
In the revision we will state this wire-width convention and the meaning of each symbol in the caption, so that the figure can be read without the text.
Thank you again for your constructive feedback, which has helped us improve the paper. Since we have addressed all of your concerns, we hope you will consider raising your rating to accept.
Thanks for your detailed reply. All my concerns are addressed. I will increase my score to 3.
We thank the reviewer and will implement all suggested changes, including adding both references.
1. Relation of Section 3.5 to [1] and [2]. We thank the reviewer for pointing us to these works.
2. Scale invariance (line 212). The equality RMS(sa)=sRMS(a) is correct for s>0 (in general, RMS(sa)=∣s∣RMS(a), since RMS is absolutely homogeneous). The alternative RMS(sa)=RMS(a) does not hold: scaling the input scales its RMS. What is scale-invariant is the normalized output, RMSNorm(sa)=RMSNorm(a), because s cancels between numerator and denominator. This is the property our proofs rely on, and the scale used there is sa=1/RMS(a)>0. To avoid the confusion, we will state both facts explicitly, write ∣s∣ in the homogeneity relation, and note the sign condition.
3. Notation in Figure 1. We will define all symbols in the caption and at first use in the text. In Figure 1, a is the input activation vector, g the learned RMSNorm weights, W the linear layer, W∗=diag(g)W the weights with g folded in, and ≜ denotes mathematical identity. We will also define the remaining symbols when they first appear.