Fair Diffusion Models via Initial-Noise Optimization in H-Space

Diffusion models can generate high-quality images, but it remains challenging to ensure fairness in the demographic composition of generated populations. Debiasing methods that optimize the initial noise before generation typically derives update directions without directly using demographic information encoded in the semantic features of denoiser, which potentially limits bias reduction or requires larger perturbations that compromise image fidelity. In this paper, we propose a new initial-noise optimization method in H-space that represents the semantic information of the denoiser. We define a batch-level fairness objective in H-space and propagate it back to the initial noise through the Jacobian, which leads to an update direction that reflects both the fairness objective and the model's feature response. Using a frozen terminal-time attribute probe, the proposed method performs prior-regularized stochastic updates on the initial noise and renders the optimized tensors with the original sampler. We provide a theoretical interpretation of this process by characterizing how initial-noise updates induce changes in H-space and establishing a gradient-alignment condition under which these updates decrease the image-space demographic distribution objective. Experiments on CelebA-HQ and FFHQ and multiple diffusion backbones, including P2, SD3, and SDXL, our method reduces fairness discrepancy by 35.8% while simultaneously improving FID and it consistently improves demographic distribution balance, demonstrating favorable fairness-fidelity trade-offs and generalization across datasets and model architectures.