Exploration–Exploitation Dilemma Introduced by Diffusion RL Fine-Tuning

Despite advances in image generation, diffusion models still struggle to directly align with specific preferences, while reinforcement learning (RL) offers an effective solution but introduces challenges in coordinating exploration and exploitation. Over-exploration reduces learning efficiency, while over-exploitation traps the policy in local optima and may even cause reward hacking. This dilemma is further aggravated in diffusion model fine-tuning by sparse rewards and long-horizon denoising. Against this backdrop, we present a systematic theoretical analysis of the coordination between exploration and exploitation in diffusion fine-tuning. First, we derive an analytical expression for the optimal policy under an uncertainty-driven exploration term. We then derive three coordination cases describing how exploration and exploitation evolve. Based on these theories, we formalize an adaptive framework named \ourmethod. Extensive 24 experiments show that our method outperforms state-of-the-art approaches in diversity (8.32%), alignment (10.30%), and speed (88.64%).