oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions

Hyper-Connections (HC) replace the single residual stream of a Transformer with $n$ parallel ones, mixing them at every layer with a learned n x n residual matrix. Leaving that matrix unconstrained places no limit on the factor by which the mixing step rescales the residual streams, and that factor compounds across layers, which destabilizes training. Manifold-constrained Hyper-Connections (mHC) address this by restricting the matrix to the doubly stochastic matrices. That caps the factor at one, so the mixing can no longer amplify any direction, but nothing bounds it from below. We prove that inside this set the mixing step can reduce the norm of the residual streams only by shrinking the differences between the streams, while their mean is left unchanged; and since this reduction compounds across layers, the streams grow increasingly alike and lose their diversity with depth. We therefore propose orthogonal Hyper-Connections (oHC), restricting the residual matrix to the rotation group $SO(n)$, so that the mixing step can neither amplify nor attenuate the residual streams in any direction, which keeps training stable and no longer contracts the differences between streams. Specifically, at the four streams used by recent HC models we parameterize the group in closed form by a pair of unit quaternions, which adds no parameters over mHC, replaces the iterative projection with a fixed pattern of signed additions, and is faster to construct. We evaluate oHC on a broad set of downstream tasks against a range of HC variants. oHC outperforms all of them, and its gain over mHC widens as model size increases.