acceptodds
Under review as a conference paper at ICLR 2027

FlashNorm: Fast Normalization for Transformers

Abstract

Normalization layers are ubiquitous in large language models (LLMs) yet represent a compute bottleneck: on hardware with distinct vector and matrix execution units, the RMS calculation blocks the subsequent matrix multiplication, preventing parallel execution. We present FlashNorm, an exact reformulation of RMSNorm followed by a linear layer that (i) eliminates the normalization weights by folding them into the subsequent linear layer, and (ii) defers the scalar RMS normalization to the output of the matrix multiplication, enabling the two operations to execute in parallel. The same techniques extend to LayerNorm and Dynamic Tanh (DyT). And due to RMS scale invariance, a linear layer between two RMSNorms makes the first one redundant, removing the pre-attention RMSNorm in QKV-normalized models (e.g., Gemma 4). In MLA models with latent normalization (e.g., DeepSeek-V2, Mistral Small 4), its weights and full-width scaling are eliminatedwhile a per-token scalar is retained on the decoupled RoPE-key path. On an NVIDIA T4 GPU, FlashNorm achieves 33-35% lower latency on the norm-then-project operation in the compute-bound (prefill) regime at SmolLM2-135M scale, and 12-14% at Llama-7B scale. We verify zero-loss weight folding on three models and validate the norm cancellation on Gemma-4-E2B, Gemma-4-12B, MiniCPM3-4B, and DeepSeek-V2-Lite. Beyond inference speed, FlashNorm simplifies model implementations by reducing parameter tensor count.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
RMSNorm, Inference, parameter efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_35ffOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes FlashNorm. Its main idea is to reorganize RMSNorm followed by a linear layer so that WaWa and RMS(a)\mathrm{RMS}(a) can be computed in parallel, thereby accelerating computation. The method is also extended to LayerNorm, DyT, and certain attention operations. The paper discusses limitations involving bias and epsilon, and presents experiments to measure the acceleration.

Critical strengths
  • The method is studied in the context of LLMs, which demonstrates its practical relevance.
  • The paper clearly and honestly discusses limitations of the method, including the treatment of bias and epsilon in RMSNorm.
Critical weaknesses

The approach in Section 3.5 appears extremely similar to the prior work in [1] and [2], but neither paper is discussed.

[1] Yuxin Guo, Yihao Yue, Yunhao Ni, Yizhou Ruan, Jie Luo, Wenjun Wu, and Lei Huang. Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm. arXiv preprint arXiv:2605.14521, 2026.

[2] Zixuan Jiang, Jiaqi Gu, Hanqing Zhu, and David Pan. Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers. Advances in Neural Information Processing Systems 36, 2023.

Questions for authors
  1. Section 3.5 does not cite [1] or [2]. Could the authors discuss how Section 3.5 relates to these works and clarify whether it is intended as a contribution of this paper?
  2. Is the equality RMS(sa)=sRMS(a)\mathrm{RMS}(sa)=s\mathrm{RMS}(a) in line 212 correct? Should scale invariance instead be expressed as RMS(sa)=RMS(a)\mathrm{RMS}(sa)=\mathrm{RMS}(a)?
  3. Could the authors explain what nn, 11, and kk denote in Figure 1? Their meanings are unclear to me from the current presentation. Symbols such as gg should also be defined when they first appear.

[1] Yuxin Guo, Yihao Yue, Yunhao Ni, Yizhou Ruan, Jie Luo, Wenjun Wu, and Lei Huang. Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm. arXiv preprint arXiv:2605.14521, 2026.

[2] Zixuan Jiang, Jiaqi Gu, Hanqing Zhu, and David Pan. Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers. Advances in Neural Information Processing Systems 36, 2023.

4 comments

We thank the reviewer and will implement all suggested changes, including adding both references.

1. Relation of Section 3.5 to [1] and [2]. We thank the reviewer for pointing us to these works.

  • [2] (Jiang et al., NeurIPS 2023) is closely related. It shows that Pre-LN transformers can be converted to RMSNorm form by absorbing the mean-centering into the preceding linear layers. We will cite it in Section 3.5 and in Related Work, and we will credit it for this idea. We will also reword the closing sentence of Section 3.5, which currently reads as if the recipe were ours.
  • [1] (Guo et al., 2026) appeared in May 2026, after the first public version of FlashNorm. We will cite it as concurrent work and discuss how it relates to Section 3.5.
  • Section 3.5 is not intended as a main contribution. It is a supporting step that lets LayerNorm models enter the FlashNorm pipeline. Our contributions are elsewhere, and we will state this in the revision:
    • Folding the norm weights into the projections post-training and pre-quantization, so that published checkpoints no longer ship separate norm-weight tensors. Inference frameworks such as vLLM cannot do this, because their inputs are already quantized models.
    • Norm cancellation (Proposition 3), which removes the pre-attention RMSNorm exactly in QKV-normalized models such as Gemma 4.

2. Scale invariance (line 212). The equality RMS(sa)=sRMS(a)\mathrm{RMS}(s\mathbf{a}) = s\mathrm{RMS}(\mathbf{a}) is correct for s>0s > 0 (in general, RMS(sa)=∣s∣RMS(a)\mathrm{RMS}(s\mathbf{a}) = |s|\mathrm{RMS}(\mathbf{a}), since RMS is absolutely homogeneous). The alternative RMS(sa)=RMS(a)\mathrm{RMS}(s\mathbf{a}) = \mathrm{RMS}(\mathbf{a}) does not hold: scaling the input scales its RMS. What is scale-invariant is the normalized output, RMSNorm(sa)=RMSNorm(a)\mathrm{RMSNorm}(s\mathbf{a}) = \mathrm{RMSNorm}(\mathbf{a}), because ss cancels between numerator and denominator. This is the property our proofs rely on, and the scale used there is sa=1/RMS(a)>0s_a = 1/\mathrm{RMS}(\mathbf{a}) > 0. To avoid the confusion, we will state both facts explicitly, write ∣s∣|s| in the homogeneity relation, and note the sign condition.

3. Notation in Figure 1. We will define all symbols in the caption and at first use in the text. In Figure 1, a\mathbf{a} is the input activation vector, g\mathbf{g} the learned RMSNorm weights, W\mathbf{W} the linear layer, W∗=diag(g)W\mathbf{W}^{\ast} = \mathrm{diag}(\mathbf{g})\mathbf{W} the weights with g\mathbf{g} folded in, and ≜\triangleq denotes mathematical identity. We will also define the remaining symbols when they first appear.

Q1: Addressed. I think the authors can add the references and adjust the descriptions in sec 3.5.

Q2: Addressed. I think the original writing may leads confusions.

Q3: How about n,1, k? I have not read about such figures before so I do not understand it. I may guess they denote the operation ×1\times 1, ×n\times n, ×k\times k, but I am not sure. Could the authors provide more detailed description about it?

Thank you for your prompt response, and for confirming that Q1 and Q2 are addressed. We will add the references, adjust the descriptions in Section 3.5, and clarify the scale-invariance statement.

Q3 (Figure 1). We apologize that the figure was not self-explanatory. The numbers nn, 11, and kk next to the slashes on the wires do not denote operations. They give the width of each wire, i.e. the number of elements it carries (the same convention as bus widths in hardware block diagrams):

  • A wire labeled nn carries a vector with nn elements: the input a\mathbf{a}, the normalized vector, and the weights g\mathbf{g}.
  • A wire labeled kk carries a vector with kk elements: the output z\mathbf{z} of the linear layer.
  • A wire labeled 11 carries a single scalar: the box "1/RMS" takes the nn elements of a\mathbf{a} and produces one number per token.

In the revision we will state this wire-width convention and the meaning of each symbol in the caption, so that the figure can be read without the text.

Thank you again for your constructive feedback, which has helped us improve the paper. Since we have addressed all of your concerns, we hope you will consider raising your rating to accept.

Thanks for your detailed reply. All my concerns are addressed. I will increase my score to 3.

Reviewer_zgKwOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a method for folding in the computation of RMSNorm into the following linear layer, a structure commonly found in e.g., pre-LN Transformers. The main contribution of the paper is showing that the learnable RMSNorm gain can be folded into the weights of the following linear layer and that the RMS reduction can run in parallel to the GEMM kernel into whose epilogue the multiplication by the inverse RMS is folded into. The authors also show that one of the RMSNorms in a RMSNorm-Linear-RMSNorm sandwich can be eliminated.

Critical strengths

The method can be applied to already trained checkpoints which makes the method useful for optimizing inference deployments.

Critical weaknesses

The main issue with this paper is the overlap with CODA [1], which the authors acknowledge in Appendix A. CODA is a generalization of this paper's main contribution in which it is shown that many memory-bound operations in the computational graph of modern LLMs can be fused into the epilogue of GEMMs. One of the operations that is folded into the epilogue in CODA is the RMSNorm application, which this paper focuses on primarily. The main additional contribution of this paper is the aforementioned observation that one can drop one of the RMSNorms in the RMSNorm-Linear-RMSNorm sandwich. This by itself does not constitute enough of a contribution for a full conference paper.

[1] https://arxiv.org/abs/2605.19269

Questions for authors

In the caption of Table 10 the authors write The proposed FlashNorm kernel is CUDA-graph captured (one cudaGraphLaunch per call); the HuggingFace and FlashInfer baselines are timed in eager mode (per-call Python dispatch). What speed ups remain once one also run the baselines with CUDA graph capture?

3 comments

We thank the reviewer and will implement all suggested changes.

  1. Overlap with CODA. We agree that CODA is closely related and will expand our discussion of it in the main text, not only in Appendix A. We also want to clarify the timeline: a preprint of this work was publicly available 2 years before CODA. The two works operate at different levels, and they compose:

    • CODA is a kernel-level technique that leaves the model unchanged. The norm weights still ship as separate tensors, and CODA must still apply them at runtime. FlashNorm is a checkpoint-level transformation. We propose folding the norm weights into the projections post-training and pre-quantization, so that published safetensors files no longer contain them. Inference frameworks such as vLLM cannot implement this folding, because their inputs are already quantized models. It must be done by the model publisher. Virtually all released LLM checkpoints on HuggingFace still ship separate norm-weight tensors, which is why we feel this should be published and promoted to make it common practice. A folded checkpoint also reduces the epilogue work that CODA-style kernels perform.

    • Norm cancellation (Proposition 3) is a substantive, independent result, not a minor addition. For QKV-normalization (RMSNorm applied to queries, keys and values), the pre-attention RMSNorm is provably redundant and can be removed entirely, with no approximation. Gemma 4 uses QKV-norm yet still computes the pre-attention norm in every block. Removing it makes QKV-norm cheaper instead of costlier, which, to our knowledge, was not previously recognized. We verify this on Gemma-4-E2B and Gemma-4-12B (perplexity change <0.001%, identical greedy continuations). No kernel-fusion system, including CODA, can obtain this, because it requires changing the computation graph and not just scheduling it.

We will revise the abstract and contributions to make this positioning explicit.

  1. CUDA graphs in Table 10. This is a fair point, and we thank the reviewer for it. The caption already discloses that the FlashNorm kernel is CUDA-graph captured while the baselines are eager. In the revision we will rerun both baselines (HuggingFace and FlashInfer) under CUDA-graph capture and report these results as the primary comparison, so that the reported speedups isolate kernel-level gains. We will present whatever speedup remains, including shapes where it vanishes.

We also note that the transformation-level results (Propositions 1-3 and the quality validation) do not depend on this prototype kernel, and the paper’s claims are scoped accordingly.

I thank the authors for their response and for addressing my critiques. I agree that the proposed method composes with CODA, and I thank the authors for clarifying the preprint timeline. While I agree that the pre-norm cancellation (Thm 3) is a useful result in and of itself, I still believe that the overall contribution is, unfortunately, not significant enough to justify a full conference paper. Hence, I will raise my score to 2.

Thank you for your response, for raising your score, and for acknowledging that our method composes with CODA and that Proposition 3 (pre-norm cancellation) is a useful result in its own right.

We understand your remaining concern is that the overall contribution is not significant enough for a full conference paper. QKV- and QK-normalization are now widely used in frontier LLMs (e.g. Gemma 4), so making them cheaper is highly relevant. We would therefore like to reframe the paper around Proposition 3 and retitle it “Fast QKV Norm”. Weight folding and deferred normalization would then be presented as supporting techniques, together with our recommendation to fold norm weights before publication.

Would this change, together with the profiling experiments and additional references already promised, be enough for you to raise your rating to accept? If not, what specifically would you need to see?

Thank you again for your time and constructive engagement.

Reviewer_Mr73OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FlashNorm, which reformulates the existing RMSNorm and its subsequent linear layer by folding the normalization weight to the linear layer's weight matrix, and deferring the RMS normalization to the output of the matrix multiplication. Such reformulation allows parallel execution of vector and matrix units, reducing the overall latency of the normalize-then-project operation. The authors show that FlashNorm can be extended to different variations of normalization layers such as LayerNorm and DyT. The evaluation shows that FlashNorm shows negligible error to the original formulation, and shows latency reduction in larger token processing.

Critical strengths
  • The authors dedicate large portion of their manuscript in thoroughly explaining the adaptability and extendability of their method to existing LLM architecture variations.
  • The authors show demonstrative figures that help understand the overall flow of the proposed reformulation.
Critical weaknesses
  • The paper presents folding normalization weights into the subsequent linear layer, which is an already established technique. As the authors have acknowledged in the related work section, such absorption of normalization weight is common in neural network quantization ([1,2]), and often used routinely to accelerate LLM execution ([3,4]). Its novelty as a standalone contribution therefore appears limited unless the authors clarify the additional technical challenges addressed in this setting.
  • The proposed method aims to provide reduced latency by parallel execution of vector and matrix units. While using separate CUDA streams can remove the software ordering constraint, it remains unsure if they guarantee perfect overlap, since actual concurrency requires sufficient unoccupied execution resources, registers, shared memory, memory bandwidth, etc. Therefore, the matrix and vector units running independently sounds more like an architectural approximation, and the useful overlap is not always guaranteed. The authors should explain how their implementation enables overlap and provide profiling evidence quantifying the achieved concurrency and latency reduction relative to sequential execution across diverse representative workloads.

[1] Raghuraman Krishnamoorthi. "Quantizing Deep Convolutional Networks for Efficient Inference: A Whitepaper." arXiv:1806.08342, 2018.

[2] Jacob et al. "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference." CVPR, 2018.

[3] Wei et al. "Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models." NeurIPS, 2022.

[4] Ashkboos et al. "QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs." NeurIPS, 2024.

Questions for authors

See weaknesses section.

3 comments

We thank the reviewer and will implement all suggested changes, including adding references [1-4].

  1. Novelty. We will cite [1-4] and state clearly that folding scales into adjacent layers is an established technique. Our contribution is different in two ways:

    • Folding as a publishing standard. In [1-4], folding is an internal step of a quantization pipeline, not something shipped in released models. If it were routine, released checkpoints would not carry separate norm-weight tensors. Yet essentially all frontier and open models on HuggingFace do. We propose folding norm weights into the projections post-training and pre-quantization, so that published safetensors files no longer contain them. Inference frameworks such as vLLM cannot implement this folding, because their inputs are already quantized models. It must be done by the model publisher. That’s why we feel this should be published and promoted to make it common practice.

    • Norm cancellation (Proposition 3), a new result. For QKV-normalization (RMSNorm applied to queries, keys and values), we prove that the pre-attention RMSNorm is redundant and can be removed entirely, with no approximation. To our knowledge this simplification has not been exploited. Gemma 4 uses QKV-norm yet still computes the pre-attention norm in every block. Removing it saves one full RMSNorm per layer and makes QKV-norm cheaper rather than costlier. We verify this on Gemma-4-E2B and Gemma-4-12B, where perplexity changes by less than 0.001% and all greedy continuations are unchanged. None of [1-4] address this.

  2. Overlap. We agree that separate CUDA streams permit overlap but do not guarantee it. In the revision we will:

    • Explain how overlap is enabled.
    • Add Nsight profiling that quantifies the achieved concurrency and the latency reduction relative to sequential execution, across several GPUs, model scales and token counts, including the regimes where overlap does not materialize.
    • Soften the wording to describe measured latency reduction and not ideal concurrency.

The benefit also does not depend on streams. Our fused Triton and CUTLASS kernels on A100 and H100 show gains from folding and fusion alone.

Reviewer_Mr73OpenReview ↗

Response to Author Rebuttal

I want to thank the authors for the rebuttal, and making further clarifications on the contributions of their paper. I appreciate the proposed revision plan, regarding norm cancellation and the proposed profiling experiments. However, my concern regarding the overall technical novelty of the work is not completely resolved.

While I agree with the practical value of performing normalization folding before model publication, I believe this contribution is more about improving deployment practices and standardizing existing techniques than to introduce a fundamentally new technical method. I also acknowledge that the norm cancellation in Proposition 3 provides an additional technical contribution distinct from conventional normalization folding. Nevertheless, I am not yet convinced that this result, together with the proposed deployment-oriented improvements, provides enough technical novelty for the paper. For these reasons, I decided to keep my original score.

Thank you for your thoughtful response, and for acknowledging both the practical value of folding before publication and the technical contribution of Proposition 3 (norm cancellation).

We noted that you are “not yet convinced” that the novelty is sufficient, and we would like to understand what would convince you. We would also like to propose a concrete change and ask for your view on it.

Proposed reframing. QKV-normalization and QK-normalization are now widely used in frontier models (e.g. Gemma 4), and making them cheaper is a worthwhile goal in itself. Proposition 3 is a new result in exactly this area: the pre-attention RMSNorm can be removed exactly in QKV-normalized models, which is a significant simplification that, to our knowledge, was not previously recognized. We could therefore reframe the paper around fast QKV- and QK-normalization, with Proposition 3 and the related QK-norm transformations as the headline technical contribution. Weight folding and deferred normalization would then be supporting techniques that make these transformations practical, together with our recommendation to fold norm weights before publication. In this framing, the new results would be the main contribution, and the known techniques would be presented as supporting tools.

Questions for you:

  • Would this reframing resolve your concern about technical novelty?
  • If not, what specifically would you need to see? For example, additional QKV/QK-norm results (more models or scales, accuracy and latency measurements), a deeper analysis of when the cancellation applies, or a different structure for the paper.

We are committed to making the paper as strong as possible and will incorporate your guidance in the revision, along with the profiling experiments, the additional references, and the clarifications already discussed.

Thank you again for your time and your constructive engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.