- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Debias Anything, a guidance-based framework for improving demographic fairness and diversity in pretrained diffusion models without requiring sensitive-attribute annotations or fine-tuning the generative backbone. The method trains an attribute-agnostic adapter to project intermediate denoising features into a pretrained vision-language embedding space, enabling semantic attribute estimation during sampling. Based on text-defined attribute directions, a batch-level optimal assignment mechanism guides generated samples toward specified demographic proportions. The authors further introduce a semantic diversity guidance term to mitigate diversity loss induced by fairness guidance. Experiments on CelebA, CelebA-HQ, and Stable Diffusion 1.5 demonstrate improved demographic balance with competitive image quality compared to several existing debiasing methods.
Critical strengths
(S1)The attribute-agnostic adapter is an interesting approach. It allows different sensitive attributes to be specified through text without training a separate attribute classifier for each one.
(S2)The results are competitive with several supervised debiasing methods, particularly on CelebA-HQ and Stable Diffusion 1.5. The paper also provides fairly extensive ablations.
Critical weaknesses
(W1)I'm not fully convinced by the claim of jointly improving fairness and diversity. The Stable Diffusion results in Table 3 report fairness and FID, but no diversity metrics or fairness-only comparison. The appendix also does not clearly specify the diversity weight used for Stable Diffusion. (W2)The proposed diversity term does not consistently outperform the alternatives. In Table 8, PerturbProj increases FID from 11.47 to 12.38 on eyeglasses, while SigLIPMS achieves a better Vendi score. It is also unclear whether the diversity gains extend to individual demographic groups rather than just the overall distribution. (W3)The generality claim seems stronger than the experimental evidence. The experiments focus entirely on human faces and uniform target proportions. There are no quantitative results for non-uniform targets or simultaneous control of multiple attributes, and the main Stable Diffusion experiments use only one random seed. (W4)The paper does not evaluate whether debiasing transfers to contexts beyond face-centric images. However, this is very important for realistic use cases.
Questions for authors
(Q1)Can the authors provide a fairness-only versus fairness-plus-diversity comparison on Stable Diffusion 1.5, including direct diversity metrics? (Q2)How does the method perform with non-uniform target proportions? Does the diversity guidance improve diversity within each demographic group?
3 comments
Thank you for the thorough response and the additional experiments. Adding the fairness-only vs. diversity comparisons on Stable Diffusion, within-group Vendi scores, non-uniform and multi-attribute evaluations, and multi-seed results directly addresses my concerns, and clarifying the face-generation scope aligns the claims well with the evidence. I will maintain my positive rating.
Thank you very much for your positive feedback and for acknowledging the additional experiments and clarifications. We are glad that our response has addressed your concerns.
Given this, we were wondering if there is any specific remaining point we could clarify that might prevent a stronger recommendation. Thank you again for your thoughtful engagement.
Response to Reviewer hoWF
We thank the reviewer for their positive and constructive review. We respond to the raised concerns below. We also invite the reviewer to review our general response above and the new revision we submitted that updates the paper with the discussed new elements.
Q1 + W1. Diversity term and metrics on Stable Diffusion
The submitted Table 3 did not use the diversity term, which is why the appendix gave no diversity weight for Stable Diffusion. As suggested, the revised Table 3 compares fairness alone with fairness + diversity on Stable Diffusion, at three seeds, with Vendi and within-group Vendi (Table 28), and CLIP-T (Table 27). The diversity term lowers FID and raises Vendi on all three attributes and at every seed, at equal or slightly lower FD, while CLIP-T barely changes. This validates our claim of jointly improving fairness and diversity.
Q2 + W3. Generality and non-uniform targets
We respond to the generality concern in the following points.
Q2 + W2. Diversity within each group
As requested, we now measure diversity within each group in every setting (Appendices D.3 and G.1). The fairness term alone reduces diversity inside the group that shrinks, and the diversity term restores part of it in almost every setting. This confirms our motivation for introducing the diversity term, and measures the effect of our method on pose collapse as asked by Reviewer quTZ.
W2. Best diversity terms
We note that PerturbProj (Eq. 12) and SigLIPMS (Appendix A.3) are both our diversity terms, and both lower FD more than the baseline terms on both attributes (Table 19). We agree, however, that no term dominates on every metric: PerturbProj has the lowest FD and MIND on gender, while on eyeglasses SigLIPMS has a better FD and Vendi, at a higher FID. We use PerturbProj in the main experiments for its FD and MIND. For diversity within each group, see Q2 + W2 above.
W4. Beyond face-centric images
All our experiments are indeed on human faces, which is the standard setting for demographic debiasing: every method we compare against evaluates it on faces. The revision makes this scope clear in the abstract and the introduction. We agree that other domains matter for real use cases, and leave them to future work. If the reviewer has a benchmark in mind, we would welcome the suggestion.