Failure Modes of the Adjoint Sampling: Instability and Slow Multimodal Adaptation

The Adjoint Sampling has been presented as a theoretically grounded approach to learning diffusion samplers from unnormalized target densities. Its existing theory characterizes the desired optimal policy as a fixed point of an idealized update, but does not establish that the proposed iteration converges to, or is locally stable around, that fixed point. We rigorously analyze its convergence using Gaussian and Gaussian-mixture targets. Their amenability to closed-form and asymptotic analysis allows us to isolate two fundamental challenges for the iterative sampler: local stability and probability redistribution across separated modes. In the Gaussian case, we prove that, over a nontrivial parameter regime, an arbitrarily small terminal-mean error alternates in sign and grows geometrically across iterations. For well-separated Gaussian mixtures, asymptotic analysis reveals two additional failure modes with exponentially slow convergence: discovery of an unrepresented mode and correction of inaccurate mode weights. These results hold with exact conditional expectations and density evolution, separating intrinsic properties of the iteration from errors due to finite sampling or neural-network approximation. Direct quadrature and Fokker–Planck solutions reproduce the predicted failure modes. Neural-network experiments show that direct velocity damping strongly suppresses the local Gaussian instability, while replay provides weaker moderation; neither intervention reliably resolves missing-mode discovery or probability redistribution across separated modes. Additional numerical experiments revealed that our identified failure modes persist beyond the idealized Gaussian setup and are also observed in higher dimensional state spaces. Together, these results show that fixed-point correctness does not imply convergence of AS method and that stabilizing local update dynamics does not by itself resolve global probability-allocation failures. Finally, we discuss the broader implications of our findings for the design and analysis of future diffusion-based sampling algorithms.