SNARL: Self-Normalizing Anchored Maximum-Entropy Reinforcement Learning

Expressive generative policies can capture complex action distributions, but maximum-entropy (MaxEnt) reinforcement learning requires evaluating the policy log-density $\log {\pi }_{\theta }(a\mid s)$, which is generally intractable for implicit generators. We introduce Self-Normalizing Anchored MaxEnt Reinforcement Learning (SNARL), an off-policy framework that learns a separate state-conditioned energy model to estimate the policy's negative log-density from samples. SNARL augments spatial and temporal score matching with an anchor derived from the Gaussian limit of variance-preserving corruption, approximately normalizing the energy model as the policy evolves. The resulting model estimates the policy's negative log-density in a single forward pass. By separating policy log-density estimation from action generation, SNARL enables maximum-entropy learning with a one-step pushforward actor that generates each action in a single network evaluation. Experiments show that SNARL is competitive with strong diffusion and flow baselines across high-dimensional continuous-control benchmarks, excels on MuJoCo Playground humanoid locomotion tasks, and provides approximately $14\times$ faster policy inference than DIME.