SwiGNLU: Nonlinear Value Bottlenecks for Parameter-Matched Gated FFNs

The feed-forward network holds most non-embedding parameters in a modern decoder, yet in SwiGLU the branch that carries its content is affine: a nonlinear gate multiplies a linear value projection. We ask whether the same feed-forward budget is better spent giving that value branch a learned nonlinear feature dictionary. SwiGNLU replaces $W_{u}x$ with $U_2\sigma (U_{1}x)$, computing $W_{\downarrow }(\sigma (W_{g}x)\odot U_2\sigma (U_{1}x))$; choosing $v_h\le dh/(d+h)$ makes the replacement no larger than SwiGLU and 0.03–1.0% cheaper in matmul FLOPs. Under one shared Muon/AdamW-on-qkv protocol, all twenty-eight completed paired comparisons favor SwiGNLU: eighteen at matched budgets, spanning 160M to 1.1B parameters on FineWeb and on independently trained Longmino models, with mean gains of 7.1–9.6 millibits per byte per cell; and ten that vary the token budget from $0.62\times$ to $9.9\times$ the Chinchilla-optimal ratio. The gain converts to 13–16% fewer tokens to reach the control's final quality, survives training to $9.9\times$ Chinchilla undiminished, transfers to held-out corpora with a larger relative margin in seven of eight cells, and moves a seven-task zero-shot battery by +0.59 points (16 of 17 pairs). Measured end to end on identical hardware, training step time matches within 1.5%, so equal quality arrives 11.8–15.2% sooner in wall-clock; the serial value path costs 2.7–5.1% per decoded token at batch 1 and shrinks with batch. Both families share the same best measured learning rate at three of four rungs, so SwiGNLU retrofits at the baseline's own tuned LR. A parameter-matched $2\times 2$ shows a consistently positive interaction between the bottleneck and its interior nonlinearity, accounting for 34–37% of the combined improvement under the qkv routing. Muon amplifies the gain without creating it: under pure AdamW at 305M, SwiGNLU wins all nine pairs at about half the margin. At 160M it also beats squared ReLU, GEGLU, and ReGLU at matched parameters. We release modules, recipe, paired run-level evidence, and complete curves.