Representational advantages of distributional reinforcement learning

Distributional reinforcement learning (dRL) agents, which learn the full distribution of returns, often outperform classical RL (cRL) agents and account for neural activity in mammalian reward-learning circuits. These findings suggest that dRL is a computationally advantageous learning strategy, although the mechanisms behind this advantage remain unclear. We develop an exact theory of learning under cRL and dRL objectives in a minimal model—a two-layer linear network—and characterize how the expectile-based dRL objective shapes state representations throughout learning. Unlike cRL, dRL learning is driven by higher-order moments of the reward distribution, which are captured by the input–output covariance matrix. As a result, dRL networks learn at least as fast as cRL networks, and differences in moments beyond the mean speed up learning further. Moments are learned sequentially, in an order set by the reward statistics, which also fully determine the geometry of the hidden representation. This representation supports faster transfer learning in dRL, whereas pre-training with the cRL objective can impede transfer. We validate our results in simulation in linear and non-linear networks. This work lays an analytic foundation for understanding the performance gains observed in more complex dRL models and tasks, and yields testable predictions for how biological circuits represent and acquire distributional information.