uHC: Uncoupled Hyper-Connections

Hyper-Connections (HC) extend Transformer residual representations into multiple streams, while Manifold-Constrained Hyper-Connections (mHC) stabilize multi-stream propagation through constrained residual mixing. However, existing HC architectures maintain coupled interfaces between residual streams and the transformation blocks: readout patterns are shared across feature dimensions, and write updates remain aligned across streams, limiting the benefits of residual-stream expansion. We identify this transformation-interface coupling as a structural bottleneck. To alleviate this limitation, we propose Uncoupled Hyper-Connections (uHC), which redesigns the transformation interfaces without modifying the underlying Transformer blocks. uHC introduces Group-Wise Feature Readout for feature-dependent stream aggregation and Low-Rank Directional Write-Back for stream-specific update directions with lightweight overhead. Across three model scales ranging from 46M to 363M parameters, uHC consistently improves validation loss and perplexity over residual connections, mHC, and mHC-lite. At the largest scale, uHC achieves the best performance across all reported metrics, reducing validation loss from 2.9470 for mHC to 2.8188. Further analyses reveal that uHC better utilizes the expanded residual-stream space and achieves more stable optimization, highlighting the importance of decoupling transformation interfaces for scaling multi-stream residual architectures.