MV-HC: Multi-View Hyper-Connections for Language Models

Hyper-Connections (HC) improve residual routing by expanding the hidden state into multiple interacting streams, while recent variants such as mHC, mHC-lite, and KromHC improve the stability and scalability of this multi-stream formulation. However, these methods typically initialize all streams from the same representation, diversifying routing but not the source information carried by each stream. We introduce Multi-View Hyper-Connections (MV-HC), which instead initializes dedicated hyper-streams from complementary tokenization views of the same input. An auxiliary representation is causally aligned to the main-token sequence and propagated through KromHC-style manifold-constrained residual mixing, while a lightweight gated reduction combines the heterogeneous streams before the LM head. We evaluate MV-HC from $60$M to $1.5$B parameters and observe consistent improvements over KromHC in validation and zero-shot downstream performance. At $1.5$B, MV-HC reduces BPB from $0.628$ to $0.617$, improves CORE from $55.2$ to $56.7$, and improves the broader language-modeling/BBH evaluation average from $35.1$ to $36.7$. These gains require only $0.14\%$ additional parameters, while training throughput remains within $1\%$ of KromHC ($62.9$k vs. $63.5$k tokens/s). These results show that complementing multi-path residual routing with multi-view source representations provides a simple and efficient way to enrich hyper-connections.