- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies stereotype bias in multimodal diffusion transformers (MM-DiTs). Using residual causal patching, the authors localize the causal influence of demographic attributes to sparse internal control layers and uncover a cross-modal bias reinforcement loop in which biased visual states feed back into the text stream. They propose FairFlow, a parameter-preserving method that learns reusable attribute directions and injects them at the control layers during early denoising. Evaluated on FLUX.1 and SD3 for gender, race, and intersectional debiasing, FairFlow improves the fairness-fidelity tradeoff.
Critical strengths
1.Overall the paper is clear, well-written, and easy to follow
2.The mechanistic analysis is substantive rather than purely empirical. The residual-causal-patching procedure, the attention metrics, and the text-trajectory projections form an evidence chain.
3.The method is simple, parameter-preserving, and reproducible: the three loss terms and the inference-time injection are precisely defined.
Critical weaknesses
1.Although FairFlow improves fairness, it incurs a modest fidelity cost. For gender debiasing on FLUX.1, the CLIP score decreases from 0.712 to 0.704 and the MUSIQ score from 0.735 to 0.725 relative to the vanilla model. Moreover, the strength ablation shows that increasing the steering strength α further reduces CLIP and MUSIQ. 2.Many closely related works (e.g., FairGen, DeCoDi, and BADGE) are neither cited nor compared, and their absence from the references and baselines weakens the novelty positioning and should be addressed.
Questions for authors
1.What about considering more combinations, e.g., three attributes (gender, age, race)? 2.What about using more than three race categories? 3.These concerns appear addressable. I would consider raising my score if the rebuttal provides convincing evidence.
5 comments
Response to Reviewer mjPD (2)
Q1: More complex attribute combinations
We conducted Gender × Race × Age experiments on both SD3 and FLUX.1 to demonstrate that FairFlow extends to more complex intersectional debiasing. We generated 12 images per prompt and report results using two age partitions:
- Two bins: under 34, and 34 or older.
- Five bins: under 25, 25–33, 34–44, 45–54, and 55 or older. In this setting, the maximum attainable fairness score is approximately 0.379 with 12 images per prompt.
These partitions provide coarse- and fine-grained age evaluations, respectively.
| Model | Method | Fairness (2 bins) | Fairness (5 bins) | CLIP | MUSIQ |
|---|---|---|---|---|---|
| SD3 | Vanilla | 0.178 | 0.096 | 0.710 | 0.744 |
| SD3 | FairFlow | 0.415 | 0.196 | 0.701 | 0.704 |
| FLUX.1 | Vanilla | 0.186 | 0.089 | 0.710 | 0.737 |
| FLUX.1 | FairFlow | 0.394 | 0.166 | 0.689 | 0.721 |
Q2: Additional race categories
We expanded the original three evaluation categories—White, Black, and Asian—to six by adding the Indian, Middle Eastern, and Latino Hispanic categories supported by DeepFace. Experiments on both models demonstrate that FairFlow extends effectively to additional attribute categories.
| Model | Method | Fairness | CLIP | MUSIQ |
|---|---|---|---|---|
| SD3 | Vanilla | 0.218 | 0.707 | 0.749 |
| SD3 | FairFlow | 0.510 | 0.703 | 0.737 |
| FLUX.1 | Vanilla | 0.208 | 0.712 | 0.735 |
| FLUX.1 | FairFlow | 0.520 | 0.700 | 0.727 |
The additional categories reuse the existing control layers and require only the optimization of three new race directions. Following the main experimental protocol, the results above use seeds 0–5, with target attributes randomly assigned according to a uniform distribution over the six race categories.
Gentle Reminder
Dear Reviewer mjPD,
As the rebuttal period draws to a close, we would appreciate any further feedback you may have on our responses. We have expanded our analysis of fidelity preservation, discussed FairGen, DeCoDi, and BADGE with additional comparisons, and added experiments on Gender × Race × Age composition and six race categories. We hope these additions address your concerns and would be pleased to provide further clarification where needed.
Thank you again for your time and thoughtful review.
Best regards, Submission 481 Authors
Response to Reviewer mjPD (1)
Thank you for the time and effort you devoted to reviewing our paper. We appreciate your positive assessment of our detailed mechanistic analysis, simple method design, and reproducibility. We address your comments below.
Intervening on demographic attributes modifies internal visual representations and also inevitably affects other aspects of the generated image. Stronger interventions generally produce larger fidelity losses, making the magnitude of these changes an important consideration. FairFlow aims to improve demographic fairness while preserving text–image alignment and image quality as much as possible. It achieves the best average fairness–fidelity tradeoff scores on both models in the main and complex scene experiments.
For example, in the gender setting on FLUX.1, fairness increases from 0.084 to 0.600, while CLIP and MUSIQ decrease only from 0.712 to 0.704 and from 0.735 to 0.725, respectively. The qualitative results in Figures 1, 16, and 17 further show that FairFlow changes demographic attributes while largely preserving the depicted subjects, occupational semantics, and scene content. The COCO-prompt experiments in Figure 19 also indicate a limited overall effect on non-target content.
The intervention-strength sweeps in Appendix O further characterize this balance: FairFlow establishes a strong fairness–fidelity Pareto frontier across the three FLUX.1 settings and in the gender and race settings on SD3. These results show that FairFlow's performance extends across intervention strengths. We recommend selecting a moderate strength according to application requirements; Appendix A provides the configurations used in our experiments.
Thank you for highlighting these relevant works. We have added comparisons with BADGE and DeCoDi on SD3 and expanded the Related Work section to discuss all three methods.
FairGen learns attribute-specific adapters attached to cross-attention blocks. DeCoDi and BADGE use output-level guidance to modify predicted noise through bias or anti-bias prompts, and BADGE additionally removes bias-direction components from text embeddings. FairFlow instead uses residual causal patching to identify control layers within MM-DiTs and learns reusable attribute directions that directly steer internal visual states at these layers. This design connects inference-time debiasing to the model's internal mechanisms for demographic control while preserving backbone parameters.
For the additional comparisons, we adapted BADGE and DeCoDi to SD3, which supports the conditional and unconditional predictions required by their CFG-based interventions. FLUX.1 natively uses a single branch with guidance modulation, so these CFG operations do not transfer directly; we therefore restrict the additional comparisons to SD3. FairGen's cross-attention adapter design likewise does not map directly to MM-DiT joint attention, so we discuss its relationship to FairFlow without including a direct adaptation in the experiments.
As shown below, FairFlow achieves the highest fairness–fidelity Pareto frontier across the three settings. We have incorporated the discussion of these methods and their architectural differences into Section 2.1 of the revised manuscript.