Gated Delta Hyper-Connections: Separating Residual Mixing from State Retention

Hyper-Connections (HC) expand Transformer residual pathways into multiple streams, offering a scaling dimension beyond model depth and width. Manifold-Constrained Hyper-Connections (mHC) stabilize cross-stream mixing, but mixing alone does not control how much of the carried state is retained: under its doubly stochastic constraint, the stream-average component is left unattenuated as layers add new outputs. Our measurements on trained models show that this component grows faster across depth than the remaining state, while write gates become more concentrated and lose mass during training. These observations motivate Gated Delta Hyper-Connections (GDHC), which adds explicit retention control alongside the existing read/write maps by combining global decay with write-aligned erasure. Decay regulates overall retention, while erasure selectively attenuates old content along the incoming write direction, independently of write strength. Experiments with mHC and identity-mixer iHC show lower training loss at both 4B- and 7B-parameter mixture-of-experts models, with 0.55B and 0.91B active parameters, respectively, and higher average downstream scores at the larger scale. Component ablations show further gains from combining decay and erasure. GDHC also flattens activation profiles and increases median write-gate mass; common-mode RMS decreases across depth on A1B mHC and becomes nearly flat on iHC. Together, these results support explicit retention as a complementary control in multi-stream residual architectures.