On the Mixing Trade-off for Expressive Hyper-Connections: A Geometric Contraction

Manifold-constrained Hyper-Connections ($\text{mHC}$) extends residual connections to multiple streams and stabilizes residual propagation by constraining stream-mixing residual matrices to the Birkhoff polytope, the set of doubly stochastic matrices. In this work, we investigate *how to effectively mix residual streams within the Birkhoff polytope*. We reveal a **mixing trade-off**: when **under-mixed** residual matrices stay close to the vertices of the Birkhoff polytope, hyper-connections will degenerate into a set of nearly independent streams; when **over-mixed** residual matrices are concentrated around the uniform barycenter of the polytope, streams will become less distinguishable from each other. We theoretically validate our findings by bounding the cumulative residual mixing, stream difference and gradient difference of mixing coefficients across layers. Motivated by these findings, we propose Contraction for Manifold-Constrained Hyper-Connections ($c\text{mHC}$), an exact doubly stochastic method which geometrically contracts the Birkhoff polytope toward its interior. This contraction mitigates under-mixing by forcing the residual matrices to explore deeper into the polytope. Moreover, we apply a moderate contraction at the shallow layers to prevent over-mixing. Extensive experiments on GPT-2 and Mixture-of-Experts at both 6B and 300B parameter scales demonstrate the effectiveness and scalability of $c\text{mHC}$. We also develop fused operators for $c\text{mHC}$ on Ascend NPUs, reducing hyper-connection execution latency by up to 44.8%. Code is available at https://anonymous.4open.science/r/cmhc.