- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an Attribute Distribution Surrogate (ADS) that predicts the demographic distribution of images generated by a text-to-image model directly from its input prompt. The method uses either the model's native text encoder or an external encoder and is evaluated on three T2I models. The authors further apply the surrogate to cross-model bias analysis and improve an existing debiasing pipeline.
Critical strengths
- A well-motivated and practical problem. Generating and labeling images for every audit prompt is costly. A prompt-only surrogate offers a promising way to scale bias auditing, potentially including API-only models.
- Promising results and practical utility. The method shows competitive results across three T2I models for both gender and race (Tables 1–2). It also improves an existing debiasing pipeline without noticeably compromising image quality (Table 3).
Critical weaknesses
- I'm not convinced the gains in Table 1 come from the proposed method itself. The Text Embedding Probe and Diffusion Activation Probe use pronoun-derived labels, while ADS is trained on distributions measured from generated images. A comparison using the same training data would help clarify how much of the improvement comes from the model rather than the supervision.
- The conclusions from the large-scale analysis seem stronger than the evidence. The 9,396 object–activity combinations are evaluated using surrogate predictions, with only 90 checked against actual generations. The TIFA analysis also lacks generation-based validation. This makes the broader findings difficult to fully trust.
- I'm also concerned about the quality of the training labels. The descriptive training set uses only one generated image per prompt, and uncertain demographic labels are discarded. How much does this affect the estimated distributions, especially for the six-category race task?
Questions for authors
1.How would the Diffusion Activation Probe perform if trained on the same prompt–distribution pairs as ADS? Would the performance gap remain? 2.Can the authors provide more generation-based validation for the object–activity and TIFA analyses, beyond the 90-prompt check already reported?
Awaiting Reviewer Response
Dear Reviewer,
We would like to thank you for your efforts and feedback. We believe our revision and responses address all concerns mentioned in the review. We would greatly appreciate getting feedback on whether there are any remaining concerns from your end.
Best, the Authors.