SphericalDEQ: An Iteration-Free Deep-Equilibrium Layer

Deep equilibrium models (DEQs) define neural layers implicitly as fixed points, enabling infinite-depth computation with constant activation memory. Their practical use, however, is hindered by the need to solve a nonlinear fixed-point problem at every forward pass and an implicit system during backpropagation. We introduce SphericalDEQ, an equilibrium layer that eliminates these iterative computations. Its key ingredient is a homogeneous spherical normalization that reduces the nonlinear fixed-point problem to a single linear solve followed by normalization. We establish conditions for the existence and uniqueness of this equilibrium and derive closed-form parameter derivatives, enabling exact backpropagation without iterative implicit differentiation, series approximations, or phantom gradients. This yields an equilibrium layer whose inference cost is comparable to that of an explicit layer once the inverse operator is cached. Experiments across image classification and neural PDE simulation show that SphericalDEQ substantially reduces the computational and memory overhead of conventional DEQs while matching or improving their predictive performance and, scales to ImageNet-1K classification and large-scale autoregressive physical simulation.