Do Hyper-Connections Transfer to Reinforcement Learning? Sphere-Constrained Mixing and the Geometry of Depth.

Hyper-connections replace residual connections with learned mixtures of all preceding layer outputs and accelerate language model pretraining, but the mixing weights must be constrained to remain stable at scale. We ask whether both the benefit and the instability transfer to on-policy reinforcement learning. We first diagnose the naive transfer: hyper-connections degrade PPO on the highest-dimensional continuous control tasks, precisely those where the gradient-routing argument predicts the largest gain, while the mixing norm drifts in every run and per-layer hidden state norms diverge at the one layer where mixing is active. To address this, we propose SHC-PPO, Sphere-Constrained Hyper connections, which keeps the mixing vector on the unit sphere via Riemannian optimization applied after each Adam step, and situate it within a design space of constraints organized by the invariant maintained, whether it holds exactly, and whether it is enforced in the forward pass or the update rule. Evaluating on MuJoCo continuous control, we find that in shallow networks every constraint holding the norm near one recovers parity with the baseline. Because shallow networks contain a single actively mixing layer, we extend the analysis to six layers, where four layers mix. There the constraints separate, and they separate on optimization geometry rather than on return: SHC-PPO preserves gradient-norm anisotropy two orders of magnitude better than unconstrained mixing, an order of magnitude better than forward renormalization maintaining the same invariant, and an order of magnitude better than a network with no mixing at all, while returns across the four conditions remain within each other's confidence intervals. Where a constraint is enforced is therefore measurable in the optimization geometry at depths where it is invisible in the reward.