Certifying Residual Architectures from Their Primitives: A Sharp Stability Threshold

Whether a deep residual architecture trains stably is usually determined by training it, which is expensive and answers the question only for the architecture, depth, and floating-point format that were tried. In practice stability is secured by heuristics on where in the residual block to place a normalization and which one to use; each is supported by experiments, and no common principle explains why and when they work. We show that stability can instead be certified before training, from the architectural primitives of the residual block, as an explicit function of depth and floating-point format. The certificate rests on two elements: (a) on a power-law growth bound for the residual block, $\Vert v(x)\Vert \le c\Vert x{\Vert }^q+b$, whose exponent $q$ is computed from the block's primitives by an arithmetic of exponents (forward); (b) on bounds on the gradients through a Lipschitz condition on the block over the reachable states (backward). The certificate determines the depth at which the state can reach the largest finite floating-point value $M$: for $q\le 1$, overflow requires at least order $\log M$ layers, whereas for $q>1$ it can occur within order ${\log }_q\log M$ layers; the threshold $q=1$ is sharp. Every normalization and every bounded activation sets $q=0$, and the arithmetic identifies minimal modifications that bring a block from $q>1$ to $q=1$ without normalization. Empirically, in forecasting only $q>1$ blocks blow up, as the forward certificate predicts; in GPT-2 on OpenWebText, $q=1$ blocks without normalization also diverge, and there the forward certificate holds while the backward one fails, in the attention block with the largest query-key coefficient. Relaxing normalization from $q=0$ to $q=1$ improves out-of-distribution generalization in operator learning, and gives accuracy gains in time-series forecasting. As recent deep models, including foundation models, grow more complex and more costly to train, our results give a way to settle the trade-off between stability and representational flexibility of a residual architecture directly from the primitives of its blocks, before any training.