acceptodds
Under review as a conference paper at ICLR 2027

Debias Anything: Fairness with Diversity without Supervision in Diffusion Models

Abstract

Although diffusion models produce high-quality images, they also reproduce and amplify demographic imbalances in their training data. Debiasing their generation process post-training w.r.t. some sensitive attribute usually relies on classifier guidance or explicit text extra-conditioning, but this reduces methods' applicability and output diversity. Conversely, methods promoting diversity alone do not ensure fair attribute representation. In this paper, we propose a method tackling fairness and diversity jointly that is generally applicable to any diffusion model and any sensitive attribute. To this end, an adapter connects the frozen diffusion model to a pretrained vision-language embedding space, enabling fairness and diversity guidance without sensitive-attribute annotations. For fairness, pairs of text prompts define attribute directions which guide batch composition towards specific proportions. For diversity, we introduce a score measuring disagreement between the semantic estimates derived from this representation. The formulation supports unconditional and text-conditional diffusion models, while requiring no prior knowledge or data of sensitive attribute. Experiments confirm that our method improves quality and diversity scores at comparable fairness levels.

Primary area:
generative models
Keywords:
Fairness, Generative Models, Diffusion, Diversity, Bias

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hoWFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Debias Anything, a guidance-based framework for improving demographic fairness and diversity in pretrained diffusion models without requiring sensitive-attribute annotations or fine-tuning the generative backbone. The method trains an attribute-agnostic adapter to project intermediate denoising features into a pretrained vision-language embedding space, enabling semantic attribute estimation during sampling. Based on text-defined attribute directions, a batch-level optimal assignment mechanism guides generated samples toward specified demographic proportions. The authors further introduce a semantic diversity guidance term to mitigate diversity loss induced by fairness guidance. Experiments on CelebA, CelebA-HQ, and Stable Diffusion 1.5 demonstrate improved demographic balance with competitive image quality compared to several existing debiasing methods.

Critical strengths

(S1)The attribute-agnostic adapter is an interesting approach. It allows different sensitive attributes to be specified through text without training a separate attribute classifier for each one.

(S2)The results are competitive with several supervised debiasing methods, particularly on CelebA-HQ and Stable Diffusion 1.5. The paper also provides fairly extensive ablations.

Critical weaknesses

(W1)I'm not fully convinced by the claim of jointly improving fairness and diversity. The Stable Diffusion results in Table 3 report fairness and FID, but no diversity metrics or fairness-only comparison. The appendix also does not clearly specify the diversity weight used for Stable Diffusion. (W2)The proposed diversity term does not consistently outperform the alternatives. In Table 8, PerturbProj increases FID from 11.47 to 12.38 on eyeglasses, while SigLIPMS achieves a better Vendi score. It is also unclear whether the diversity gains extend to individual demographic groups rather than just the overall distribution. (W3)The generality claim seems stronger than the experimental evidence. The experiments focus entirely on human faces and uniform target proportions. There are no quantitative results for non-uniform targets or simultaneous control of multiple attributes, and the main Stable Diffusion experiments use only one random seed. (W4)The paper does not evaluate whether debiasing transfers to contexts beyond face-centric images. However, this is very important for realistic use cases.

Questions for authors

(Q1)Can the authors provide a fairness-only versus fairness-plus-diversity comparison on Stable Diffusion 1.5, including direct diversity metrics? (Q2)How does the method perform with non-uniform target proportions? Does the diversity guidance improve diversity within each demographic group?

3 comments
AuthorsOpenReview ↗

Response to Reviewer hoWF

We thank the reviewer for their positive and constructive review. We respond to the raised concerns below. We also invite the reviewer to review our general response above and the new revision we submitted that updates the paper with the discussed new elements.

Q1 + W1. Diversity term and metrics on Stable Diffusion

The submitted Table 3 did not use the diversity term, which is why the appendix gave no diversity weight for Stable Diffusion. As suggested, the revised Table 3 compares fairness alone with fairness + diversity on Stable Diffusion, at three seeds, with Vendi and within-group Vendi (Table 28), and CLIP-T (Table 27). The diversity term lowers FID and raises Vendi on all three attributes and at every seed, at equal or slightly lower FD, while CLIP-T barely changes. This validates our claim of jointly improving fairness and diversity.

Q2 + W3. Generality and non-uniform targets

We respond to the generality concern in the following points.

  • Non-uniform targets. As detailed in the general response, we conducted a new comparison on CelebA-HQ with the skewed targets used in Balancing Act's paper (80% men, 90% Black). Our method comes close to both targets, with the lowest FID of all compared methods (Appendix D.5, Table 10). Score Guidance gets as close or closer, but at a much higher FID, and Balancing Act stays far from the 90% target at every scale we tried.
  • Multiple attributes. We also conducted a new comparison on CelebA-HQ with a joint gender x race target, 25% for each of the four groups, against Balancing Act and Score Guidance. Our method comes much closer to the target than every baseline (Appendix D.6, Table 11).
  • Seeds. Tables 2 and 3 now report standard deviations computed with three seeds, for every method in Table 2 and for ours in Table 3, whose baselines are copied from DiffLens (see the general response). On Stable Diffusion, the submitted single-seed FD and FID are within one standard deviation of the three-seed means, so the conclusions do not change.
  • Faces-focused evaluation. See W4.

Q2 + W2. Diversity within each group

As requested, we now measure diversity within each group in every setting (Appendices D.3 and G.1). The fairness term alone reduces diversity inside the group that shrinks, and the diversity term restores part of it in almost every setting. This confirms our motivation for introducing the diversity term, and measures the effect of our method on pose collapse as asked by Reviewer quTZ.

W2. Best diversity terms

We note that PerturbProj (Eq. 12) and SigLIPMS (Appendix A.3) are both our diversity terms, and both lower FD more than the baseline terms on both attributes (Table 19). We agree, however, that no term dominates on every metric: PerturbProj has the lowest FD and MIND on gender, while on eyeglasses SigLIPMS has a better FD and Vendi, at a higher FID. We use PerturbProj in the main experiments for its FD and MIND. For diversity within each group, see Q2 + W2 above.

W4. Beyond face-centric images

All our experiments are indeed on human faces, which is the standard setting for demographic debiasing: every method we compare against evaluates it on faces. The revision makes this scope clear in the abstract and the introduction. We agree that other domains matter for real use cases, and leave them to future work. If the reviewer has a benchmark in mind, we would welcome the suggestion.

Thank you for the thorough response and the additional experiments. Adding the fairness-only vs. diversity comparisons on Stable Diffusion, within-group Vendi scores, non-uniform and multi-attribute evaluations, and multi-seed results directly addresses my concerns, and clarifying the face-generation scope aligns the claims well with the evidence. I will maintain my positive rating.

Thank you very much for your positive feedback and for acknowledging the additional experiments and clarifications. We are glad that our response has addressed your concerns.

Given this, we were wondering if there is any specific remaining point we could clarify that might prevent a stronger recommendation. Thank you again for your thoughtful engagement.

Reviewer_quTZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Debias Anything, a sampling-time guidance method that sets the proportions of a sensitive attribute in the outputs of a frozen diffusion model. The attribute is specified only by short text prompts. The method has three components. The first is a simulation-free semantic adapter: a small network, trained once per generator and independently of any attribute, that maps the denoiser's bottleneck activation to the SigLIP 2 embedding of the clean image with a cosine regression loss. The second is a batch-level fairness term. Each attribute value is described by a sentence and compared to a neutral sentence, and the normalised difference of the two text embeddings defines a logit; a Hungarian assignment with integer quotas then decides which image is pushed towards which value at each step. The third is a diversity term (PerturbProj), which moves SGMS (Um & Ye, 2024) into the embedding space and measures diversity as the cosine disagreement between the adapter's readout on the noisy sample and on a re-noised, re-denoised Tweedie estimate. The method works with both unconditional and text-conditional models. The authors report fairness/quality trade-offs that match or beat label-supervised baselines on CelebA-HQ and Stable Diffusion 1.5.

Critical strengths

S1. A general, label-free debiasing method that holds up against supervised baselines: One adapter per generator, attributes named in free text at sampling time, no fine-tuning of the generator, and support for both unconditional and text-conditional models. That combination is useful in practice, and Table 1 shows that each prior method is missing at least one part of it. FairGen, for example, learns attribute-specific adapters, while the adapter here does not depend on the attribute. The CelebA-HQ numbers support the claim. In the fairness setting the method reaches FD 0.008 / 0.008 / 0.065 (gender / race / eyeglasses) at FID 50.7 / 58.2 / 59.8. Score Guidance, which is supervised, only gets to a similar FD at FID 64.7 to 66.8, and Balancing Act, trained on labels, stays at FD 0.043 to 0.187. On Stable Diffusion 1.5, FD falls from about 0.56 to 0.75 without guidance to about 0.20 on all three attributes, with no attribute supervision. The method can also steer towards attributes that have no label at all (red hair in Figure 3, the compound target in Figure 5), which label-based methods cannot do. This is the main reason I lean towards acceptance.

S2. A well-motivated way to read semantics from noisy states, with experiments to back it: Regressing the embedding of the clean image is simple, needs no simulation, and avoids encoding blurry, off-manifold Tweedie averages. The paper tests this directly. Figure 4 (AUC vs. σ), Table 5 and Figure 6 show that on the skewed eyeglasses attribute the pixel/Tweedie readout stalls at FD 0.489 while MIND and FID get worse, whereas the h-space readout reaches FD 0.156 and MIND keeps improving up to w = 200. The authors also say where the advantage goes away: on gender the two readouts are within seed noise. I expect this idea to be useful outside fairness too, for example for semantic control, auditing, or catching biased trajectories early on any frozen generator.

Critical weaknesses

W1. The experiments support less than the paper claims, especially on diversity: The paper's positioning rests on "fairness with diversity" and on parity with supervised methods. As it stands, I find the fairness part convincing, but not yet the diversity part or the parity claim on SD.

  • Diversity in the main comparisons: Diversity is in the title, and the abstract says the method "improves quality and diversity scores at comparable fairness levels". Yet Tables 2 and 3 report only FD and FID. There is no Vendi, precision/recall or coverage, even though the setup lists these metrics. On CelebA-HQ the diversity term changes FID by 0.0 to 0.7 in the fairness setting (50.7→50.0, 58.2→58.0, 59.8→59.8) and by 1.5 to 2.0 in the quality setting, from one seed and with no variance reported. The paper does not say whether the diversity term is used on Stable Diffusion at all. In the five-seed CelebA ablation (Table 8), the semantic diversity terms raise FID on eyeglasses (11.47→12.38 for PerturbProj), and the MIND changes are within one standard deviation. The pose collapse shown in Figure 1 is never measured, and the paper does not study how the fairness and diversity terms interact at high weights.
  • Stable Diffusion claims: The text concludes that the unsupervised results are "better than or comparable to" supervised methods. On gender and age, though, the method's FD (0.200, 0.206) is about four times that of DiffLens (0.046, 0.049), and on race Latent direction is lower (0.175). What the table does show is that this is the only method reasonably balanced on all three attributes. That is a narrower claim. The FID gaps to the copied baseline numbers (113.8 vs. 112.8 on gender, 114.7 vs. 116.9 on race) are smaller than the authors' own reproduction gap for the unguided model (117.1 reproduced vs. 120.1 reported).
  • Statistical support and scope: The CelebA-HQ and SD results come from a single seed, so I cannot judge several of the comparisons in Section 5.1, where FID differs by 0.6 to 2.2 points. "Arbitrary proportions" is one of the main selling points in Table 1, but every experiment uses uniform targets, and every experiment is on faces.

W2. "Without supervision" is weakened by how the method is tuned and trained:

  • The guidance weights are specific to each attribute and sensitive. On CelebA-HQ, w_fair is 20 / 100 / 500 in the fairness setting and 5 / 30 / 180 in the quality setting, w_div drops to 75 for race, and the limitations section admits a "narrow useful range that we select by sweeps". Choosing these weights means measuring FD, and measuring FD needs an attribute classifier, that is, labels, possibly from the same pipeline that produces the reported numbers. A fully label-free deployment would need a label-free way to set w_fair, such as the adapter's own zero-shot estimate of the proportion. The paper does not show how results change when the weight is picked without labels.
  • The SD adapter is trained on 50k images generated from prompts built on the same four occupations used for evaluation. This is not attribute supervision, but it is tuning on the evaluation prompts. "Trained once per generator" implies that one adapter works for unseen prompts, and the paper does not test that.
  • The theory says little about the regime the method runs in. The bounds in Lemma 1 carry factors e^{±2w}, and at the reported weights (w_fair from 5 up to 3,000) they are vacuous. The lemma therefore only shows that the gap goes to zero as V_k → 0, without a useful rate. The logits are also computed with the unit-norm adapter output instead of m, so the linearity S_k(x_σ) = E[S_k(x_0) | x_σ] that the lemma relies on does not hold for the quantity being guided: normalisation rescales the logits by 1/‖m‖, which grows at high noise. Proposition 1 is correct but elementary. It also does not cover what happens at assignment switch points, where the gradient is undefined and re-solving the assignment at every step could make images oscillate between targets.

W3. Missing evaluations for practical use (lower priority than W1 and W2):

  • Prompt fidelity on SD 1.5: Guidance changes the content of text-conditioned samples, but no text-image alignment metric such as CLIP score is reported. I cannot tell whether fairness costs prompt adherence.
  • Training-free text-side baselines: On SD 1.5, recent inference-time methods that edit the prompt or the text embedding without retraining are natural baselines for a training-free method.
Questions for authors
  1. Weight selection: How exactly were w_fair and w_div chosen for CelebA-HQ and SD? On which images, with which classifier, and were they selected on the same samples and metric reported in Tables 2 and 3? Could you report results with a weight chosen by a label-free criterion (e.g., the adapter's own predicted proportion), or with one weight shared across attributes on CelebA-HQ, as you did on SD?

  2. Diversity in the main comparisons: Is the diversity term active in Table 3? Could you add Vendi and precision/recall (or density/coverage) to Tables 2 and 3, and report several seeds with standard deviations for Table 2? Measuring the pose collapse shown in Figure 1 would directly support the diversity claim.

  3. Why does the diversity term lower FD? In Table 8 (gender), PerturbProj brings FD from 0.031 down to 0.006, and Table 2 shows similar drops. I would not expect a term with no attribute information to improve parity. Is it interacting with the assignment, for example by pushing ties across the quota boundary, and does this still happen with non-uniform targets? In the other direction, do f_fair and f_div conflict at high weights?

  4. Baselines on CelebA-HQ: How is race predicted for Balancing Act, given that CelebA-HQ has no race label? Score Guidance was run with its authors' gender preset on all three attributes. Did you try tuning it per attribute, as you did for your method? Balancing Act and the unguided model use batch size 100 and your method uses 50. Does batch size affect FD under exact quotas?

  5. Race reference set: The FID reference set for race is built from CLIP ViT-B/32 similarity to "a black person", and your guidance uses a SigLIP text direction for the same sentence. Could this favour a method driven by text embeddings? An FID against a reference set selected with FairFace would rule this out.

  6. Adapter generalisation on SD: How does the adapter do on prompts outside the 20 training prompts, such as other occupations or scenes that are not portraits?

  7. Lemma 1 in practice: Can the bound be stated with the observed range of S_k, which is much smaller than [-1, 1], so that it means something at the weights you use? And how does normalising the adapter output affect the argument?

  8. Factual check: Appendix E says yours is the lowest race discrepancy in Table 3, but Latent direction reports 0.175 against your 0.195. Please correct or clarify.

Further questions (helpful for the final version):

  1. Prompt fidelity: Could you report CLIP text-image alignment, or a similar metric, for the SD 1.5 experiments with and without guidance?

  2. Encoder choice: How much do FD, FID and stability change if SigLIP 2 is replaced with another encoder (e.g., CLIP ViT-L/14)?

  3. Compute: How much does it cost to train the adapter, and how much slower is sampling per image, with and without the diversity term? How does the per-step Hungarian assignment scale with batch size? Do you cache or approximate it across steps?

  4. Assignment stability: How often does an image's assigned value change during sampling? Do switches near the quota boundary cause oscillation, and do you use any smoothing?

  5. Diversity hyperparameters: How are σ_s and the other parameters of f_div chosen, and how sensitive are the results to them?

  6. Classifier-free guidance scale: On SD 1.5, does the fairness you reach hold across CFG scales?

Suggestions (future work, not affecting my score): non-uniform targets, and joint or intersectional targets over several attributes, especially since D.2 reports that steering eyeglasses shifts gender (0.096→0.343); a variant for single-sample or streaming generation, where there is no batch to couple (e.g., a running buffer of past assignments); a DiT or flow-matching backbone, to support the "any diffusion model" claim; and a domain other than faces.

3 comments
AuthorsOpenReview ↗

Response to Reviewer quTZ (1/2)

We thank the reviewer for their positive and constructive review. We respond to the raised concerns below. We also invite the reviewer to review our general response above and the new revision we submitted that updates the paper with the discussed new elements.

We follow the reviewer's question ordering and tag the relevant weakness for each question.

Q1 + W2. Label-free weight selection

On CelebA-HQ, we tuned the weights using the FD computed with evaluation classifiers; note that our revision includes results with two new seeds which give the same conclusions. The reviewer correctly points out that this may introduce indirect attribute supervision. We thank the reviewer for proposing to test a fully label-free criterion. In our updated manuscript, we included a new experiment testing a label-free weight selection rule in Appendix D.1.

As suggested, we use our own zero-shot classifier on images to compute FD and measure FID against real images balanced by that classifier. We select the smallest fairness weight whose FID is no worse than without guidance and within one standard error of best FD. This picks the paper's weights on race and eyeglasses, and on gender a smaller weight that is as fair, with a slightly better FID (Tables 5 and 6, Appendix D.1). Our method can therefore be made fully label-free, including the choice of its weights.

Note that the raised issue "race weight lowered to 75" in Appendix B.6 was an error in our text: all our runs actually use the fairness weight of the fairness setting (100) with wdiv=40w_{\mathrm{div}} = 40. This has been corrected in the revision.

Q2 + W1. Diversity in the main comparisons + seeds

We would like to thank the reviewer for the insightful suggestions. Our revision includes new experiments addressing all requests.

  • Added diversity term in Table 3, and diversity scores in Tables 2 and 3. Table 2 now reports Vendi and coverage, with precision, recall and density in Appendix D.2 (Table 7), and the diversity scores of Table 3 are in Appendix G.1 (Table 28). We prefer coverage to recall, which rewards outlier images (Naeem et al., 2020): in our runs, Balancing Act has the highest recall but the lowest precision on every attribute (Appendix D.2), while our quality setting has the highest coverage. Overall, the diversity term raises Vendi and lowers FID in almost every setting, at little or no cost in fairness.
  • Multiple seeds. We now report standard deviations over three seeds in Tables 2 and 3.
  • We measure pose collapse as a loss of diversity within a class (Appendix D.3, Table 8; Appendix G.1, Table 28, within-group Vendi). The fairness term alone reduces diversity inside the class that shrinks, and the diversity term restores part of it, notably at the quality setting. This measures the effect of our method on pose collapse as requested.

Q3. Why does the diversity term lower FD?

As the reviewer suggests, by perturbing the images, it can help some of those that miss their assigned class reach it. This effect is clear on CelebA (Table 19). The fairness and diversity terms do conflict at high diversity weights: FID rises and FD can worsen (Tables 17 and 18).

Q4. Baselines on CelebA-HQ

  • Race for Balancing Act is predicted by its own h-space classifier, trained on CLIP pseudo-labels (Parihar et al., 2024, supp. D.1). Every row of Table 2 is then scored with the same FairFace classifier.
  • Score Guidance with its authors' per-attribute settings does not change the comparison: it matches our eyeglasses FD at a much higher FID, and its gender and race FD get worse than with the preset (now clarified in Appendix B.4).
  • New results in the revision show that batch size barely changes FD under exact quotas (Appendix D.7, Table 13).

Q5. CLIP-independent race reference set

To test whether the FID reference set built with CLIP favors text embedding-based methods like ours as hypothesized, we include in our updated paper new experiments with a reference set picked by FairFace in Appendix D.8 (Table 14). Our method now obtains the best race FID, while it was behind Balancing Act with the CLIP-picked one: Balancing Act is indeed possibly biased toward CLIP as its race predictor is trained on CLIP pseudo-labels. Therefore, the CLIP-based reference set does not favor our SigLIP-based guidance method.

Q6 + W2. The SD adapter and unseen prompts

We include in the revision a new experiment on four occupations outside the training prompts (CEO, housekeeper, librarian, pilot; Appendix G.2, Table 30). We show that our adapter generalizes, as guidance balances them as well as the training ones.

AuthorsOpenReview ↗

Response to Reviewer quTZ (2/2)

Q7 + W2. Theory

  • Lemma 1 in practice. The exponential factor comes from bounding the clean-image logit by [−1, 1]. A tighter bound would require the logit to stay in a smaller range almost surely, which is a restrictive assumption.
  • Normalization. Our cosine loss learns the direction of mm, as stated in Section 4.1. We clarify in Section 4.2 that the lemma analyzes the unnormalized mean mm and does not directly bound the normalized classifier or its guidance gradient. For each noisy image, using the direction of mm amounts to the weight w/∥m∥w/\lVert m \rVert, so the lemma holds with that weight, and Proposition 1 holds for normalized logits.
  • Switch points. In our revision, we show that assignments switch only in the first steps, never late in sampling, and nearly all images end in their assigned class (Appendix D.9, Table 15).

Q8 + W1. Claims on Stable Diffusion

  • "Better than or comparable to supervised methods." We agree that Table 3 supports only the narrower claim the reviewer states, and revised it accordingly (Section 5.1 and Conclusion): our method is the only one reasonably balanced on all three attributes, with the lowest worst-attribute FD.
  • Factual check (Q8). Latent direction indeed has the lowest race FD. We thank the reviewer for spotting this and corrected Appendix G.
  • FID gaps smaller than our reproduction gap. This gap came from a mistake in Appendix B.4, which we correct in our revision: the previously given FID of 117.1 for our unguided reproduction was in fact the FID of one of our guided runs. Our unguided runs actually give 120.8 ± 1.7 at three seeds, against 120.1 reported by DiffLens, so the reproduction is validated and our FIDs can be compared with the copied baseline numbers. Compared this way, at three seeds, our FID equals the best baseline's on gender, is lower on race and higher on age; with the diversity term, it is lower on gender and race.

Q9 + W3. Missing evaluations

  • Prompt fidelity. Our guidance actually barely changes CLIP-T, as shown in the original submission (now Table 27), so our method preserves prompt adherence.
  • Training-free text-side baselines. In the revision, we test the training-free ethical intervention of Bansal et al. (2022), which appends "if all individuals can be a [occupation] irrespective of their [gender/age/race]" to the prompt, in Table 3. This baseline balances attributes much less than our guidance.

Further questions

  • Q10. Encoder choice. Another encoder means training a new adapter. We consider this as future work.
  • Q11. Compute. Training the adapter takes 3 to 11 h on one A100 (Table 4). On CelebA, the diversity terms add up to about 43% to the sampling time (Appendix E.4, Table 20). The assignment is solved exactly at each step and costs little next to a network evaluation.
  • Q12. Assignment stability. See Q7, "switch points". We use no smoothing.
  • Q13. Diversity hyperparameters. σs\sigma_s is swept on CelebA (Appendix E.5). On SD, FD stays at the fairness-only level when making the diversity weight vary between 0 and our final choice (Appendix G.1, Table 29).
  • Q14. Classifier-free guidance scale. We conducted new experiments demonstrating that the fairness gain holds at CFG scales 5, 7.5 and 10, with a better FID at each scale; at a fixed fairness weight, the gain only shrinks slightly as the scale grows (Appendix G.3, Table 31).
  • Suggestions. We added non-uniform targets (cf. general response and Appendix D.5) and a joint gender-race target (Appendix D.6).
Reviewer_quTZOpenReview ↗

Reply

Thank you for all your responses. All the modifications made in the text improved its readability and made it a very nice manuscript. I will keep my scores and evaluation.

Reviewer_aAMyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a training-free guidance method for fair generation in diffusion models. The authors steer a frozen model to produce chosen proportions of a sensitive attribute, and then the attribute is described in text. Then they use a lightweight adapter that maps the U-Net's internal mid-block features to a vision-language embeddings for zero-shot attribute estimation. A batch-level assignment pushes each image toward its nearest attribute value at each step so the quotas come out exactly, and they add a rarity-based term to keep the samples diverse. They use CelebA datasets with stable diffusion 1.5 for major experiments, showing that fairness achievement is competitive with supervised baselines.

Critical strengths
  • The paper is well organized and easy to follow
  • The motivation is crystal clear and well-articulated. Existing debiasing methods suffer from at least one of four major limitations. The authors consider all of them. The train-free set up is appealing in real-world applications to reduce the cost to achieve fair models
  • Using the Hungarian algorithm at every step to dynamically match batch trajectories looks a novel application in this problem
  • Strong experiment results
Critical weaknesses
  • "Debias Anything" looks over-sold since all experiments use faces and uniform targets. I don't think the current experiments can justify this is a "debias anything" method. Models and datasets are limited to support this title
  • I am not an expert in diffusion models, is Stable Diffusion 1.5 a recommended baseline in this area of studies? What would results be with newer implementations?
  • The diversity term seems not working very well if looking at FID changes on CelebA-HQ at the fairness setting? For CelebA-HQ and Stable Diffusion tables, there is no diversity metrics.
Questions for authors

see above

2 comments
AuthorsOpenReview ↗

Response to Reviewer aAMy

We thank the reviewer for their positive and constructive review. We respond to the raised concerns below. We also invite the reviewer to review our general response above and the new revision we submitted that updates the paper with the discussed new elements.

1. The scope of "Debias Anything"

"Debias Anything" looks over-sold since all experiments use faces and uniform targets. [...] Models and datasets are limited to support this title

The title refers to the following properties: debiasing, for any attribute, any frozen diffusion model toward any target proportions. We clarify this in the introduction. We verified the "any attribute" and "any frozen diffusion model" parts in our initial submission.

  • Any attribute. The attribute is given only as short sentences, and a single adapter per generator, trained without attribute labels, serves every attribute. We can steer any attribute such as "red hair", which CelebA-HQ has no label for (Figure 3).
  • Any frozen diffusion model. The generator is never retrained, and the same adapter recipe and guidance run on unconditional and text-conditional models, in pixel and latent space (EDM on CelebA, P2 on CelebA-HQ and SD 1.5; cf. next point).

All these experiments are on faces, the domain in which every baseline we compare against evaluates demographic debiasing. We clarify this in the revision, in the abstract and introduction. However, the reviewer rightly points out that we did not check how our method performs on non-uniform attribute targets.

As suggested, in our revision, we conducted a new experiment with non-uniform targets in Table 10 on CelebA-HQ with the skewed targets used in Balancing Act's paper (80% men, 90% Black). Our method comes close to both targets, with the lowest FID of all compared methods. Score Guidance gets as close or closer, but at a much higher FID, and Balancing Act stays far from the 90% target at every scale we tried.

Therefore, our title is justified. That said, it is meant as a short, memorable name for the method, and we are happy to change it if it matters for the reviewer's assessment.

2. Relevance of Stable Diffusion 1.5

is Stable Diffusion 1.5 a recommended baseline in this area of studies? What would results be with newer implementations?

Stable Diffusion 1.5 remains the common testbed for debiasing text-to-image models. It is the model in DiffLens's protocol (CVPR 2025), on which all six baselines of Table 3 are evaluated, including the most recent, ICM (CVPR 2026).

A comparison on a newer model would be interesting, but requires substantial additional work: three of our baselines have no released code (ours will be released), and others release components tied to SD 1.5 (our method is not). Our adapter recipe and guidance already run on three different generators, unconditional or text-conditional, in pixel or latent space, and the adapter is directly applicable to activations of newer U-Nets (SDXL) and transformers (SD 3, FLUX).

We expect our method to transfer, and possibly to work better, since guidance can only steer toward images the model is able to generate. As noted in our discussion of limitations, on Stable Diffusion 1.5, the same gender guidance brings doctors to 50% men, but nurses and receptionists, which Stable Diffusion 1.5 almost never generates as men, only to about a quarter. This may be alleviated in newer, more performant models.

3. Diversity term and diversity metrics

The diversity term seems not working very well if looking at FID changes on CelebA-HQ at the fairness setting? For CelebA-HQ and Stable Diffusion tables, there is no diversity metrics.

As requested, we added diversity metrics to both tables in our revision. Table 2 now reports Vendi and coverage, with precision, recall and density in Appendix D.2 (Table 7), and Table 3 adds the diversity term, with Vendi and within-group Vendi in Appendix G.1.

We agree that the effect of the diversity term is small at the fairness setting of Table 2, where the fairness guidance weight is high. At the quality setting, however, with a lower fairness weight, and on Stable Diffusion, the term consistently lowers FID and raises Vendi, at little or no cost in fairness. The fairness term alone keeps overall diversity near the unguided level, because balancing adds variety between classes, but it reduces diversity inside the class that shrinks; the diversity term restores part of it, as we show with our new experiment (Appendix D.3, Table 8).

Thanks for your response. I will keep my positive rating

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to reviews

We would like to thank the reviewers for their positive and constructive reviews. They find the method general and practical (one attribute-agnostic adapter per generator, generic attributes named in text, no attribute labels), well motivated, with strong results that are competitive with supervised baselines.

They share similar concerns about evidence and metrics for diversity, lack of multi-seed studies, absence of non-uniform fairness targets in the experiments, and restriction to human faces. We address these shared concerns below, notably with new experiments, and in the attached revision (changes highlighted in red).

  • Diversity term on SD and diversity metrics (all reviewers). Table 2 now reports Vendi and coverage; precision, recall and density are added in Appendix D.2, Table 7. Table 3 adds the diversity term on SD, with Vendi and within-group Vendi analyzed in Appendix G.1. Overall, the diversity term raises Vendi and lowers FID in almost every setting, at little or no cost in fairness, which validates this contribution.
  • Non-uniform targets (all reviewers). We conducted a new comparison on CelebA-HQ with the skewed targets used in Balancing Act's paper (80% men, 90% Black). Our method comes close to both targets, with the lowest FID of all compared methods (Appendix D.5, Table 10). Score Guidance gets as close or closer, but at a much higher FID, and Balancing Act stays far from the 90% target at every scale we tried.
  • Diversity within each group & pose collapse (hoWF, quTZ). New experimental results show that the fairness term alone does reduce diversity inside the group that shrinks, but that the diversity term restores part of it (Appendix D.3, Table 8; Appendix G.1, Table 28, within-group Vendi). This confirms our motivation for introducing the diversity term, and measures the effect of our method on pose collapse as requested by Reviewer quTZ.
  • Multiple seeds (quTZ, hoWF). We added standard deviations for three-seed runs to Table 2 for all compared methods, and to Table 3 for our method only (since DiffLens reports only one run for all baselines). All previous conclusions still hold.
  • Human faces (aAMy, hoWF). All our experiments are indeed on human faces, which is a standard setting for fairness research: every method we compare against does the same. The revision makes this scope clear in the abstract and introduction; other domains are left to future work.

Detailed responses as well as answers to other concerns raised by a single reviewer, including the corrections they asked for in the revision, are provided individually.

AuthorsOpenReview ↗

Thanks & new results on images other than human faces

We would like to thank again the reviewers for their positive feedback to our rebuttal. We are glad to share a new revision of the paper including new results that we believe could strengthen their favorable assessment, on images other than human faces (aAMy, hoWF).

We conducted experiments on Shen et al. (2024)'s setup with group, sports and scene pictures. To our knowledge, this is the only public benchmark with released code, prompts and models that evaluates demographic debiasing beyond images of human faces. We report in Appendix H overall similar fairness performance and better diversity compared to Shen et al. (2024) (tie on groups, better on scenes, worse on sports), while, unlike the baseline, requiring no attribute supervision, generator retraining and specialization in text-to-image (Table 1).

We have not updated accordingly the claims in our abstract and limitations yet. We will do so in the camera-ready version, should our paper be accepted, if the reviewers acknowledge these new results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.