Reference-Centered Generalization Bounds for Modern Pre-Normalized Decoders: Explicit Constants and a Vacuity Threshold
Abstract
We derive an explicit reference-centered uniform generalization bound for pre-normalized decoders, including grouped-query attention, rotary maps, RMS normalization, and SwiGLU, and ask when such a bound can be nontrivial at all. The bound has two moving parts: a transport coefficient, which carries an early parameter perturbation through the residual stack, and a covering-entropy term. We prove a sharp threshold for the normalized stage coefficient of the truncated-Dudley objective: at or above it, the optimized certificate must equal its range cap, and in a stronger regime improvements in stage-cover constants drop out of the objective altogether. On a frozen public 1.1B checkpoint, a norm-matched Jacobian audit gives a median transport of for the product of separately estimated sublayer norms but only for the directly estimated composite. Three SmolLM2 checkpoints (135M–1.7B parameters) reproduce the separation, with median paired gaps of – decades against on the 1.1B reference; all operator norms in this audit are finite-search lower estimates. We find that, within this covering-and-transport proof structure, the obstruction sits in stagewise transport rather than in the cover constants, and our results point to direct, direction-aware composition bounds rather than further tightening of stagewise constants.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.