FlashNorm: Fast Normalization for Transformers

Normalization layers are ubiquitous in large language models (LLMs) yet represent a compute bottleneck: on hardware with distinct vector and matrix execution units, the RMS calculation blocks the subsequent matrix multiplication, preventing parallel execution. We present FlashNorm, an exact reformulation of RMSNorm followed by a linear layer that (i) eliminates the normalization weights by folding them into the subsequent linear layer, and (ii) defers the scalar RMS normalization to the output of the matrix multiplication, enabling the two operations to execute in parallel. The same techniques extend to LayerNorm and Dynamic Tanh (DyT). And due to RMS scale invariance, a linear layer between two RMSNorms makes the first one redundant, removing the pre-attention RMSNorm in QKV-normalized models (e.g., Gemma 4). In MLA models with latent normalization (e.g., DeepSeek-V2, Mistral Small 4), its weights and full-width scaling are eliminatedwhile a per-token scalar is retained on the decoupled RoPE-key path. On an NVIDIA T4 GPU, FlashNorm achieves 33-35% lower latency on the norm-then-project operation in the compute-bound (prefill) regime at SmolLM2-135M scale, and 12-14% at Llama-7B scale. We verify zero-loss weight folding on three models and validate the norm cancellation on Gemma-4-E2B, Gemma-4-12B, MiniCPM3-4B, and DeepSeek-V2-Lite. Beyond inference speed, FlashNorm simplifies model implementations by reducing parameter tensor count.