CLOSER: Closing the Feedback Loop for Inference-Time Scaling of Diffusion Models

Inference-time scaling aims to improve diffusion-model outputs by allowing more time and computation during generation. Using task feedback to direct this computation requires turning a terminal assessment into an effective correction. Our controlled diagnostics show the challenge: informative evaluator gradients can still yield harmful updates, and the effective intervention stage varies across objectives. We therefore introduce CLOSER, a training-free controller that turns terminal feedback into verified, position-aware state corrections. At an initial or intermediate sampler state, CLOSER differentiates through the remaining discrete sampler to construct a bounded candidate set. It executes these candidates through the unchanged suffix and accepts a correction only after terminal evaluation, keeping the incoming trajectory available for rollback. After acceptance, refreshed feedback guides the next proposal from the updated sampler state. This separation lets gradients guide the search while terminal criteria determine which trajectory is retained. Across SD3.5-Medium and Z-Image on DPG-Bench, OneIG-EN, and GenEval, CLOSER achieves the strongest reported terminal utility across all evaluated backbone–benchmark settings, exceeding the strongest GenEval baselines by 0.113 and 0.106. The evaluated operating points also outperform the compared utility–cost baselines. Position analysis connects these gains to correction scale: early interventions favor structural changes, whereas late interventions favor preference and alignment refinement in the controlled study.