Inside the Residual Stream: Neuron-Level Mixing with Nonlinear Low-Rank Branches

Redesigning residual connections can improve the way information is propagated across deep Transformer layers. Methods such as Hyper-Connection and Attention Residuals enrich connectivity by introducing interactions among different hidden states. However, these approaches primarily operate at the level of whole hidden states, leaving finer-grained interactions within individual states largely unexplored. Frac-Connection explores finer-grained connectivity by partitioning hidden states into multiple streams and applying structured interactions, but its interaction pattern is constrained by a fixed coordinate-wise decomposition. In this work, we propose **ResLorB** (**Res**idual Connections with Nonlinear **Lo**w-**r**ank **B**ranches), which introduces a lightweight nonlinear low-rank transformation at the block input to enable learnable neuron-level interactions. With only minimal additional parameters and computational overhead, ResLorB consistently improves model performance without requiring specialized optimization or infrastructure support. It is naturally compatible with existing multi-state architectures, including mHC, and the low-rank branch can further serve as an additional interactable state to enable richer interaction patterns. Extensive experiments on DeepSeek-V3-style models scaling up to 3B/9B parameters demonstrate consistent improvements over strong baselines, including a 0.011 reduction in training LM loss and 1.4% improvement in downstream average performance at the 9B scale, while introducing only 1% additional parameters and negligible latency overhead. On 3B-scale models, ResLorB improves downstream performance by approximately 1% on average. Combining ResLorB with existing multi-state architectures further yields non-saturating gains and consistent improvements.