DAHC: Differentiation-Augmented Hyper-Connections via Structured Transformations

Hyper-Connections (HC) expand the Transformer residual state into multiple parallel streams, yet the resulting streams need not become correspondingly differentiated representations. We show that, in mHC, stream differentiation emerges gradually during training and is reorganized non-monotonically across network depth. Motivated by these observations, we propose Differentiation-Augmented Hyper-Connections (DAHC), which introduces structured sources of stream differentiation both at stream creation and throughout depth. DAHC combines Differentiated Stream Expansion (DSE), which replaces replicated initialization with stream-indexed causal transformations, and Layerwise Differentiation Augmentation (LDA), which constructs depth-adapted stream candidates and integrates them through state-conditioned write-back while preserving the native mHC pathway. Empirically, DAHC reaches the final mHC training loss and validation perplexity after 7.3k and 7.5k updates, requiring 27% and 25% fewer updates, respectively. At the matched 10k-update budget, DAHC further reduces validation perplexity from 29.34 to 28.49, corresponding to a 2.90% improvement over mHC.