Posterior Gradient Estimation Changes the Capabilities of Intrinsically Driven Agents

Changing only posterior gradient estimation in world models with discrete latent states can lead intrinsically driven agents to acquire markedly different capabilities. We investigate this effect by changing how gradients are estimated through posterior samples across intrinsic-drive methods. Experiments across methods and tasks reveal substantial changes in performance, with both the magnitude and direction depending on the drive and task. Policies with similar aggregate scores can also acquire markedly different skill profiles. These findings show that posterior gradient estimation affects both performance and the skills acquired under intrinsic rewards.