Anchored Manifold Alignment Stabilizes Bilevel Refinement for Long-tailed Diffusion

Diffusion models often suffer from degraded tail category generation when pro cessing long-tail distribution data due to extreme sample imbalance. Although Bilevel Optimization has recently been introduced for downstream task align ment in diffusion models, existing frameworks are predominantly designed for training from scratch. This approach readily induces calibration drift when fine tuning pre-calibrated models, undermining established tail category semantics. This paper proposes a phased decoupled fine-tuning framework specifically de signed for long-tail diffusion models. The core innovation lies in introducing the Static Pool Resampling (SPR) mechanism: during upper-layer objective evalua tion, resampling from a fixed anchor sample pool generated in the pre-calibration stage (Stage 1) transforms unconstrained sampling optimization into manifold constrained distribution feature alignment. Theoretical and experimental evidence demonstrates that this "resampled proxy evaluation" maintains statistical consis tency with hyper-gradient properties even under extreme long-tail distributions, ensuring both task-specific alignment and distributional integrity. This discovery provides crucial technical support for diffusion model fine-tuning. Results across multiple benchmark long-tail datasets demonstrate that our method enhances task performance while effectively mitigating calibration bias, validating its robustness and superiority under complex long-tail distributions. The code is provided in the supplementary material.