acceptodds
Under review as a conference paper at ICLR 2027

Fair Diffusion Models via Initial-Noise Optimization in H-Space

Abstract

Diffusion models can generate high-quality images, but it remains challenging to ensure fairness in the demographic composition of generated populations. Debiasing methods that optimize the initial noise before generation typically derives update directions without directly using demographic information encoded in the semantic features of denoiser, which potentially limits bias reduction or requires larger perturbations that compromise image fidelity. In this paper, we propose a new initial-noise optimization method in H-space that represents the semantic information of the denoiser. We define a batch-level fairness objective in H-space and propagate it back to the initial noise through the Jacobian, which leads to an update direction that reflects both the fairness objective and the model's feature response. Using a frozen terminal-time attribute probe, the proposed method performs prior-regularized stochastic updates on the initial noise and renders the optimized tensors with the original sampler. We provide a theoretical interpretation of this process by characterizing how initial-noise updates induce changes in H-space and establishing a gradient-alignment condition under which these updates decrease the image-space demographic distribution objective. Experiments on CelebA-HQ and FFHQ and multiple diffusion backbones, including P2, SD3, and SDXL, our method reduces fairness discrepancy by 35.8% while simultaneously improving FID and it consistently improves demographic distribution balance, demonstrating favorable fairness-fidelity trade-offs and generalization across datasets and model architectures.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion, H-Space, Jacobian, Fair Generation, Initial-Noise Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_CPnFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the control of demographic distributions, such as gender and race, in collections of diffusion-generated images. It proposes an initial-noise optimization method in H-space and evaluates it on two P2 models trained on CelebA-HQ and FFHQ.

Critical strengths
  • The method does not require updating the generative model’s weights. Population-level distribution control is a worthwhile research problem.
  • Compared with vanilla generation, the method substantially reduces FD in some settings.
Critical weaknesses
  • The manuscript does not include the AI usage disclosure section
  • All five experiments in Table 4 remain pending.
  • SD3 and SDXL are mentioned in the abstract but are not evaluated in the reported experiments.
  • The advantage over baselines is inconsistent. For example, Table 1 reports a mean FD of 0.015 for rollout optimization, compared with 0.018 for the proposed method.
Questions for authors
  • Where are the SD3 and SDXL experiments? How is the 35.8% reduction reported in the abstract calculated, and which settings and baselines does it refer to?
  • At matched output FD, does the method offer advantages in FID, diversity, or content preservation?
  • Is the submitted manuscript an unfinished version? Several experimental results and verification fields are explicitly marked as pending.
Reviewer_56fLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a method for improving demographic balance in diffusion models by optimizing the initial noise. It uses an attribute probe on the denoiser's H-space features to define a batch-level fairness objective and backpropagates the gradient to the initial noise. The optimized noise is then passed through the original sampler. The paper also provides a theoretical analysis of the induced feature changes and reports experiments on face generation.

Critical strengths

1.Optimizing initial noise through semantic features is an interesting approach to controlling demographic distributions without modifying the pretrained generator. 2.The paper considers both fairness and image quality and includes comparisons with H-space guidance, different attribute evaluators, and several optimization variants.

Critical weaknesses

1.The claims in the abstract do not match the reported experiments. The abstract mentions SD3 and SDXL and claims improvements in both fairness and FID, but the completed experiments only cover P2 models. In Table 2, the full method actually increases FID from 43.5 to 46.5 compared with vanilla generation.

2.The main theoretical motivation lacks experimental validation. The analysis relies on gradient alignment between the H-space objective and the output fairness objective, but this alignment is never measured. Table 4 lists several key comparisons as pending, including gradient alignment and matched-FD evaluations.

3.The advantage over existing methods is still unclear. The comparisons are not conducted at matched fairness levels, and the method is more computationally expensive than in-loop distribution guidance (2.4× versus 1.4× vanilla runtime). The current results do not establish a better fairness–quality trade-off.

Questions for authors

1.Can the authors clarify the discrepancy between the abstract and the experiments, particularly the SD3/SDXL results and the claimed FID improvement?

2.Can the authors provide the missing gradient-alignment measurements and matched-FD comparisons from Table 4? These seem essential to support the proposed mechanism.

Reviewer_W6MhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an inference-time method to control the attribute distribution of samples generated by a diffusion model by optimizing the initial batch of Gaussian noise, before running a sampler like DDIM. Particularly, the authors train a linear attribute probe on the U-Net bottleneck features at the highest-noise timestep. Particularly, during the initial optimization phase, the method selects a subset of overrepresented samples near the probe's decision boundary, computes a batch-level distribution matching loss, and backpropagates this objective through the U-Net encoder at timestep TT to the initial noise via a vector-jacobian product. This way, the initial noise is updated as per Eq. 6. Once optimized, the stored initial noise are passed through the sampler to generate final samples.

Critical strengths
  1. The method by construction decouples the "noise optimization" and "sampling". This helps in debiasing without affecting the inference time which is usually not the case with inference-time optimization methods. Further, the optimized initial noise can be cached and deterministically replayed which is another aspect that is not common in this area of research.
  2. The paper clearly articulates the difference between updating directly the bottleneck activation space v/s updating the input space. This is helpful because it helps in understanding that the later induces a sensitivity-weighted feature update and coherently recomputes both the bottleneck and all skip-connections from a single valid input tensor.
Critical weaknesses
  1. The submission reads completely drafted by AI; while this is not a bad thing, the draft is very hard to read through. There are several places where new definitions pop out of nowhere, some of the claims mentioned in abstract are not present in text at all, some of the tables are left incomplete, etc.
  2. The abstract claims results on CelebA-HQ, FFHQ and multiple diffusion backbones like P2, SD3 and SDXL. However, in paper, there no experiments on SD3 or SDXL.
  3. Table 4 and Table 5 have completely blank result columns!
  4. Section 4.4 and Appendix F are literally an unexecuted experimental plans!
  5. The noise at the first denoising step is pure noise, hence the SNR is ~00. While U-Net bottleneck features (hh-space) carry rich semantic information at intermediate timesteps, I am unsure of any meaningful feature a the first denoising step.
  6. While there is no additional cost added at each denoising step. The initial optimization itself (50 steps) doubles the overall optimization steps (50 DDIM + 50 initial noise optimization).
Questions for authors

Could you explain the reasons for points 1, 2, 3 and 4 in weaknesses above?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.