FASE: Factor-Aware Spectral Evolution for Gradient-Free Continuation of Diffusion Post-Training

Reward-based diffusion post-training methods such as Flow-GRPO and DiffusionNFT rely on backpropagation through the generator. We propose Factor-Aware Spectral Evolution (FASE), a gradient-free continuation stage that starts from a gradient-trained, reward-adapted DiffusionNFT low-rank adaptation (LoRA) and performs no further backpropagation. Our design tests a constrained continuation hypothesis motivated by spectral diagnostics on Stable Diffusion 3.5 Medium (SD3.5-M): measured changes in $A$'s row space and $B$'s column space decrease early while ordered singular values continue to evolve, and the two factors exhibit unequal magnitudes of spectral change under the training parameterization. FASE instantiates this hypothesis by freezing the specific handoff SVD bases and optimizing their singular values using separable CMA-ES. A factor-aware adaptive controller reallocates exploration between the two factors using whitened CMA-ES mean steps while preserving total coordinate variance. Starting from task-specific DiffusionNFT checkpoints, FASE reaches GenEval $0.98$, OCR $0.98$, and PickScore $24.25$; reference target scores from Flow-GRPO and complete DiffusionNFT runs trained from the base model provide endpoint context, not a matched from-scratch comparison. Controlled ablations isolate the adaptive factor-allocation controller by comparing it with uniform exploration and a fixed non-uniform $A$/$B$ allocation from the same warm-start under matched population and generation budgets. Experiments across GenEval, OCR, and PickScore show that FASE effectively continues reward-adapted diffusion models without further backpropagation, improving all three target rewards and achieving the best observed endpoints among matched spectral-continuation variants.