ReCache: Learning Budget-Aware Caching Schedule for Diffusion Models via REINFORCE

Modern diffusion models generate high-quality images and videos, but their iterative denoising process makes inference expensive. Feature caching accelerates sampling by reusing or predicting intermediate activations across diffusion steps, exploiting computational redundancy along the reverse trajectory. In this work, we align caching schedule selection with the goals of inference: preserving fidelity to uncached outputs and improving generation quality under a fixed budget of full model evaluations. We propose **ReCache**, a policy-gradient method that learns a budget-conditioned scheduling policy to select which diffusion steps to fully recompute. Training combines an output-fidelity objective with a generation-quality reward, requiring no labeled data or backpropagation through the full denoising trajectory. ReCache supports both feature reuse and feature forecasting; for each model and caching mechanism, a single trained policy adapts across computational budgets at inference time without retraining. On FLUX.1-dev, ReCache reduces LPIPS by 31% compared to DiCache at a $5.04\times$ FLOPs reduction. On Wan2.1, at a $3.50\times$ FLOPs reduction, it reduces LPIPS by 45% and improves VBench by 6.77 points over uniform HiCache.