ITO-HC: Identity-Centered Tanh Off-Diagonal Hyper-Connections

Hyper-Connections (HC) widen the residual state and learn dynamic read, write, and stream-mixing maps. Manifold-Constrained Hyper-Connections (mHC) stabilize these maps by constraining the residual mixer to the Birkhoff polytope, but repeated nonnegative mixing can suppress stream-difference directions and finite Sinkhorn iterations introduce projection error. We propose ITO-HC, an identity-centered, input-independent residual mixer that adds bounded, signed off-diagonal corrections while preserving an exact unit diagonal. ITO-HC retains the mHC-style dynamic pre/post routers and nonlinear Transformer branches, combining an explicit residual highway with controlled cross-stream interactions. We derive a layer-attention formulation that exposes the low-rank dense routing structure of the identity-residual baseline, and we establish conditional perturbation and spectral stability bounds for ITO-HC. Experiments on OpenWebText with GPT-2-style models compare ITO-HC with identity, mHC, mHC-lite, and a single-stream residual baseline across multiple stream widths. The results support the proposed stability–capacity trade-off while highlighting the need for further scale and routing-diagnostic studies.