Equilibrium Updates for Residual Learning

Multi-stream residual networks combine mechanisms for retaining intermediate state and injecting sublayer output. When mixing and writing are independently parameterized, changing retention also changes the state maintained by repeated input. This paper introduces Equilibrium-coordinate Hyper-Connections (eHC), which parameterize residual writes via this conditional equilibrium using the existing write head. Given a doubly stochastic mixing matrix $P$, eHC writes $g1+(I-P)q$, sharing a gain $g$ and zero-mean coordinates $q$. This separates additive accumulation of the stream mean from target-centered disagreement dynamics, making the direct sensitivity to mixing dependent on target error. For fixed positive mixing, equilibrium coordinates are in one-to-one correspondence with signed affine writes, so eHC changes the learning of equivalent writes without changing their local expressive capacity. Controlled experiments with language models indicate improved language modeling and zero-shot transfer compared to mHC and a matched signed-write baseline with little additional training cost. Gradient ablations identify benefits of adaptive write coordinates and their gradient coupling to mixing, and retention interventions reveal that preserving the target reduces predictive sensitivity. These results suggest that parameterizing residual writes by the states they should sustain, rather than learning injection independently of retention, may improve residual networks.