acceptodds
Under review as a conference paper at ICLR 2027

FairFlow: Demystifying and Mitigating Stereotype Bias in Text-to-Image Diffusion Transformers

Abstract

Multimodal diffusion transformers (MM-DiTs) underpin modern text-to-image systems, yet can reproduce demographic bias under neutral prompts. Existing debiasing methods were largely designed for U-Net backbones and offer limited guidance on where semantic attributes are controlled in joint transformer architectures. In this work, we trace stereotype bias through MM-DiTs and find that its causal influence is concentrated in a sparse set of semantic control layers. These layers exhibit strong visual and cross-modal signals, as revealed by our attention analysis. Crucially, our investigation into text-state trajectories uncovers a pivotal bias reinforcement loop, demonstrating how repeated text-image updates systematically amplify initial bias priors during the generation process. Based on these observations, we propose FairFlow, a parameter-preserving method that learns reusable attribute directions and injects them into visual features at the identified control layers during early denoising. Experiments on FLUX.1 and Stable Diffusion 3 across multiple demographic settings show that FairFlow improves the fairness–fidelity tradeoff and remains effective on complex scene prompts.

Primary area:
generative models
Keywords:
Text-to-Image Model, Mechanistic Interpretability, Debiasing, Fairness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mjPDOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies stereotype bias in multimodal diffusion transformers (MM-DiTs). Using residual causal patching, the authors localize the causal influence of demographic attributes to sparse internal control layers and uncover a cross-modal bias reinforcement loop in which biased visual states feed back into the text stream. They propose FairFlow, a parameter-preserving method that learns reusable attribute directions and injects them at the control layers during early denoising. Evaluated on FLUX.1 and SD3 for gender, race, and intersectional debiasing, FairFlow improves the fairness-fidelity tradeoff.

Critical strengths

1.Overall the paper is clear, well-written, and easy to follow

2.The mechanistic analysis is substantive rather than purely empirical. The residual-causal-patching procedure, the attention metrics, and the text-trajectory projections form an evidence chain.

3.The method is simple, parameter-preserving, and reproducible: the three loss terms and the inference-time injection are precisely defined.

Critical weaknesses

1.Although FairFlow improves fairness, it incurs a modest fidelity cost. For gender debiasing on FLUX.1, the CLIP score decreases from 0.712 to 0.704 and the MUSIQ score from 0.735 to 0.725 relative to the vanilla model. Moreover, the strength ablation shows that increasing the steering strength α further reduces CLIP and MUSIQ. 2.Many closely related works (e.g., FairGen, DeCoDi, and BADGE) are neither cited nor compared, and their absence from the references and baselines weakens the novelty positioning and should be addressed.

Questions for authors

1.What about considering more combinations, e.g., three attributes (gender, age, race)? 2.What about using more than three race categories? 3.These concerns appear addressable. I would consider raising my score if the rebuttal provides convincing evidence.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mjPD (1)

Thank you for the time and effort you devoted to reviewing our paper. We appreciate your positive assessment of our detailed mechanistic analysis, simple method design, and reproducibility. We address your comments below.

W1: Decreases in image fidelity

Intervening on demographic attributes modifies internal visual representations and also inevitably affects other aspects of the generated image. Stronger interventions generally produce larger fidelity losses, making the magnitude of these changes an important consideration. FairFlow aims to improve demographic fairness while preserving text–image alignment and image quality as much as possible. It achieves the best average fairness–fidelity tradeoff scores on both models in the main and complex scene experiments.

For example, in the gender setting on FLUX.1, fairness increases from 0.084 to 0.600, while CLIP and MUSIQ decrease only from 0.712 to 0.704 and from 0.735 to 0.725, respectively. The qualitative results in Figures 1, 16, and 17 further show that FairFlow changes demographic attributes while largely preserving the depicted subjects, occupational semantics, and scene content. The COCO-prompt experiments in Figure 19 also indicate a limited overall effect on non-target content.

The intervention-strength sweeps in Appendix O further characterize this balance: FairFlow establishes a strong fairness–fidelity Pareto frontier across the three FLUX.1 settings and in the gender and race settings on SD3. These results show that FairFlow's performance extends across intervention strengths. We recommend selecting a moderate strength according to application requirements; Appendix A provides the configurations used in our experiments.

W2: Discussion of and comparisons with FairGen, DeCoDi, and BADGE

Thank you for highlighting these relevant works. We have added comparisons with BADGE and DeCoDi on SD3 and expanded the Related Work section to discuss all three methods.

FairGen learns attribute-specific adapters attached to cross-attention blocks. DeCoDi and BADGE use output-level guidance to modify predicted noise through bias or anti-bias prompts, and BADGE additionally removes bias-direction components from text embeddings. FairFlow instead uses residual causal patching to identify control layers within MM-DiTs and learns reusable attribute directions that directly steer internal visual states at these layers. This design connects inference-time debiasing to the model's internal mechanisms for demographic control while preserving backbone parameters.

For the additional comparisons, we adapted BADGE and DeCoDi to SD3, which supports the conditional and unconditional predictions required by their CFG-based interventions. FLUX.1 natively uses a single branch with guidance modulation, so these CFG operations do not transfer directly; we therefore restrict the additional comparisons to SD3. FairGen's cross-attention adapter design likewise does not map directly to MM-DiT joint attention, so we discuss its relationship to FairFlow without including a direct adaptation in the experiments.

As shown below, FairFlow achieves the highest fairness–fidelity Pareto frontier across the three settings. We have incorporated the discussion of these methods and their architectural differences into Section 2.1 of the revised manuscript.

MethodGender: Fairness ↑Gender: CLIP ↑Gender: MUSIQ ↑Race: Fairness ↑Race: CLIP ↑Race: MUSIQ ↑Gender × Race: Fairness ↑Gender × Race: CLIP ↑Gender × Race: MUSIQ ↑Average fairness ↑Average tradeoff ↑
DeCoDi0.1540.7010.7520.1990.7030.7490.1620.7010.7510.1720.293
BADGE0.4460.7090.7340.7780.7060.7240.4400.6980.6870.5550.695
FairFlow0.6370.6980.7440.7410.6990.7310.4900.6780.7080.6230.756
AuthorsOpenReview ↗

Response to Reviewer mjPD (2)

Q1: More complex attribute combinations

We conducted Gender × Race × Age experiments on both SD3 and FLUX.1 to demonstrate that FairFlow extends to more complex intersectional debiasing. We generated 12 images per prompt and report results using two age partitions:

  • Two bins: under 34, and 34 or older.
  • Five bins: under 25, 25–33, 34–44, 45–54, and 55 or older. In this setting, the maximum attainable fairness score is approximately 0.379 with 12 images per prompt.

These partitions provide coarse- and fine-grained age evaluations, respectively.

ModelMethodFairness (2 bins)Fairness (5 bins)CLIPMUSIQ
SD3Vanilla0.1780.0960.7100.744
SD3FairFlow0.4150.1960.7010.704
FLUX.1Vanilla0.1860.0890.7100.737
FLUX.1FairFlow0.3940.1660.6890.721

Q2: Additional race categories

We expanded the original three evaluation categories—White, Black, and Asian—to six by adding the Indian, Middle Eastern, and Latino Hispanic categories supported by DeepFace. Experiments on both models demonstrate that FairFlow extends effectively to additional attribute categories.

ModelMethodFairnessCLIPMUSIQ
SD3Vanilla0.2180.7070.749
SD3FairFlow0.5100.7030.737
FLUX.1Vanilla0.2080.7120.735
FLUX.1FairFlow0.5200.7000.727

The additional categories reuse the existing control layers and require only the optimization of three new race directions. Following the main experimental protocol, the results above use seeds 0–5, with target attributes randomly assigned according to a uniform distribution over the six race categories.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer mjPD,

As the rebuttal period draws to a close, we would appreciate any further feedback you may have on our responses. We have expanded our analysis of fidelity preservation, discussed FairGen, DeCoDi, and BADGE with additional comparisons, and added experiments on Gender × Race × Age composition and six race categories. We hope these additions address your concerns and would be pleased to provide further clarification where needed.

Thank you again for your time and thoughtful review.

Best regards, Submission 481 Authors

Reviewer_ZXvAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies demographic bias in text-to-image diffusion transformers and proposes FairFlow to reduce such bias. The authors use residual causal patching to identify layers related to demographic attributes and analyze how interactions between text and image features may reinforce bias during generation. Based on this analysis, FairFlow learns attribute directions and applies them to selected visual features without modifying the original model parameters. Experiments on FLUX.1 and Stable Diffusion 3 show improvements in demographic balance across gender, race, and intersectional settings.

Critical strengths

1.The paper first analyzes the internal layers of MM-DiTs before introducing the debiasing method. The results in Section 4 suggest that some layers have a stronger influence on demographic attributes than others, which provides a basis for the proposed intervention.

2.The method does not require updating the backbone parameters. The authors also compare different layer selections and direction construction methods in the ablation experiments, which helps explain the contribution of these components.

3.The experiments are conducted on both FLUX.1 and SD3, covering gender, race, and their combination. The comparisons with FairDiffusion, FairImagen, and other methods show that FairFlow can improve demographic balance while generally preserving image quality.

Critical weaknesses

1.In Section 4.1, the authors mainly use LPIPS to select the control layers. However, higher similarity to the target image does not necessarily indicate a successful change in the intended demographic attribute. Changes in background, clothing, or other visual features may also affect the score. If the authors could provide an attribute-level evaluation of the patched images, the selection of control layers would be more convincing.

2.The results in Tables 1 and 2 show improvements in fairness, but some settings still have relatively low fairness scores. For example, the intersectional fairness score on SD3 is 0.490, and the results on complex-scene prompts are not always the best among the compared methods. The paper could provide more discussion of these cases and explain when the proposed intervention is less effective.

3.The paper claims that FairFlow supports adjustable target demographic ratios, but the main experiments mainly focus on uniformly sampled target attributes. This does not fully show how accurately the method can achieve different target distributions. It would be useful to report the actual generated proportions under several non-uniform target ratios.

Questions for authors

1.Would the selected control layers change if demographic attribute classification accuracy were used instead of LPIPS in Section4.1?

2.Is there any evidence that the cross-modal bias reinforcement observed in SD3 also occurs in FLUX.1?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZXvA (1)

Thank you for your time and thoughtful review. We appreciate your positive assessment of our systematic layer localization, attribute-direction construction, and comprehensive ablation studies. We address your comments below.

W1 / Q1: Additional attribute-level evaluation of patched images

We added an attribute-level causal score based on DeepFace in Figure 9, using the target-minus-source class logit difference to evaluate attribute changes in patched images, and compared it with the original LPIPS-based localization. The attribute-level evaluation corroborates LPIPS-based localization on FLUX.1. On SD3, LPIPS better identifies control layers that enable effective attribute control through joint intervention.

The two models exhibit different patterns:

  • FLUX.1: The two metrics identify highly consistent control layers. The visualizations in Figure 8 also show that single-layer patching can produce clear attribute changes while preserving recognizable image content.
  • SD3: The two metrics differ substantially. Some layers with the highest DeepFace scores produce severely distorted faces (Figure 10). For these samples, high attribute-classification scores cannot reliably be interpreted as successful attribute transfer. By contrast, the control layers identified by LPIPS produce weaker attribute changes under single-layer patching, while our main experiments demonstrate that their joint intervention effectively changes demographic attributes.

We have refined the discussion in Section 4.1 of the revised manuscript and present LPIPS-based localization, attribute-level evaluation, and visual inspection as complementary evidence.

W2: Relatively low fairness scores in some settings

Some settings are challenging because models respond differently to interventions, independently learned attribute directions can interact after composition, and complex scenes require transfer across diverse visual contexts. Although FairFlow does not achieve the highest fairness in every setting, it achieves the best or second-best fairness scores across the evaluated settings and the best average fairness–fidelity tradeoff across attribute settings in each evaluation scenario. We discuss the possible reasons for the remaining gaps below.

First, the models respond differently to interventions. In the main experiments, SD3 yields some second-best fairness results, whereas FLUX.1 consistently achieves the highest fairness. Our additional patching experiments show that a single control layer can produce clear attribute changes in FLUX.1, while SD3 relies more on multi-layer cooperation. FairFlow currently applies interventions uniformly across the control layers. Better coordination across these layers may further improve performance.

Second, independently learned attribute directions may interfere with one another. FairFlow optimizes each attribute direction separately and combines them linearly at inference time. A direction that is effective in a single-attribute setting may interfere with other directions after composition. For intersectional debiasing, disentangling the directions and combining orthogonalized components may help reduce such interference.

Third, our attribute directions are learned only from portrait-oriented prompts, yet remain effective when transferred to complex scene prompts, showing stronger generalization than the evaluated baselines. For example, FairImagen and WeakGuidance exhibit substantial decreases in fairness on FLUX.1 complex scenes relative to the main experiments, whereas FairFlow’s average fairness increases slightly, by 0.026. Some settings nevertheless remain suboptimal. Complex scenes involve diverse environments, poses, and viewpoints, which may limit the effectiveness of applying the same direction at a fixed strength to every image token. A potential extension is to use layer responses to adaptively identify person-related regions and assign different intervention strengths to different image tokens.

We have added this analysis and discussion to the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer ZXvA (2)

W3: Additional results with different target group ratios

We added five non-uniform target distributions across gender and race on both SD3 and FLUX.1. The tables below report the generated distributions, target-attribute hit rates, and corresponding Vanilla results.

FairFlow substantially improves target-attribute hit rates in every setting: from 20.74%–57.19% to 75.42%–82.80% on SD3, and from 18.54%–60.03% to 54.93%–83.59% on FLUX.1. The generated distributions also shift in accordance with the target ratios.

SD3:

AttributeTarget ratioGenerated distribution (%)Vanilla hit rate ↑FairFlow hit rate ↑
Gender1:235.56 / 64.4451.00%82.80%
Gender2:155.93 / 44.0749.33%80.13%
Race4:1:163.71 / 13.88 / 22.4157.19%81.44%
Race1:4:131.44 / 48.49 / 20.0720.74%75.42%
Race1:1:429.10 / 13.55 / 57.3621.91%80.43%

FLUX.1:

AttributeTarget ratioGenerated distribution (%)Vanilla hit rate ↑FairFlow hit rate ↑
Gender1:228.09 / 71.9150.68%83.59%
Gender2:148.73 / 51.2751.02%77.50%
Race4:1:176.15 / 9.03 / 14.8260.03%75.81%
Race1:4:151.19 / 34.35 / 14.4618.54%58.67%
Race1:1:456.63 / 9.01 / 34.3521.94%54.93%

Q2: Verifying cross-modal bias reinforcement in FLUX.1

We added visual residual patching and FairFlow steering experiments on FLUX.1, using both bias-axis projection and cosine-similarity difference to measure bias in text states. The results confirm that FLUX.1 exhibits cross-modal bias reinforcement consistent with the trend observed in SD3, and that FairFlow can effectively disrupt this bias reinforcement loop (Appendix E, Figures 11-12).

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer ZXvA,

As the rebuttal period draws to a close, we would appreciate any further feedback you may have on our responses. We have added attribute-level validation of control-layer selection, experiments with non-uniform target ratios, and cross-modal feedback analyses on FLUX.1, together with a discussion of the remaining limitations in challenging settings. We hope these additions address your concerns and would welcome the opportunity to discuss any points that remain unclear.

Thank you again for your time and constructive feedback.

Best regards,
Submission 481 Authors

Reviewer_tCbTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FairFlow learns attribute-specific vectors for demographic steering in FLUX.1 and SD3 while keeping the backbone frozen. The vectors are applied at layers identified through residual patching during early denoising. Experiments on held-out occupations show improved average fairness and fidelity tradeoffs under the paper’s metrics, with additional complex-scene evaluations and design ablations. The paper also studies visual-to-text interactions as a possible mechanism of bias reinforcement.

Critical strengths
  • FairFlow achieves the best average fairness and fidelity tradeoff among the evaluated methods on both backbones under the reported metrics. The improvements come with relatively modest changes in CLIP and MUSIQ. Evaluation on 100 occupations excluded from optimization supports generalization beyond the training occupations, and the complex-scene results provide further evidence beyond face-centered prompts. The paper appropriately acknowledges settings where a baseline performs better.
  • The SD3 ablations support the main design choices. Selected layers outperform equally sized random and bottom-layer sets in fairness, optimized directions outperform direct feature differences, and the gender loss ablation examines the role of each objective. The learned vectors are reusable, and Table 3 reports low additional SD3 inference time once offline preparation is complete.
Critical weaknesses
  • The mechanistic analysis supports visual-to-text influence more clearly than bias amplification. Figures 4 and 7 show that visual interventions shift text representations toward the bias or anti-bias reference, but do not establish whether this feedback increases demographic bias in the generated images. Since the reinforcement loop is presented as a main contribution, the distinction between influencing text states and amplifying output bias needs clearer justification.
  • The reliability of the main fairness comparisons is not sufficiently established. Fairness is computed from only six images per occupation before averaging across occupations, which is particularly limited for the six-group intersectional setting. Appendix L provides useful agreement checks for the attribute evaluator, but the main results lack uncertainty estimates or an analysis of sensitivity to sampling seeds. Appendix K also provides useful strength sweeps, although the criterion for choosing the configurations reported in the main tables remains unclear.
  • The computational evaluation covers inference overhead but leaves the offline preparation cost unreported. FairFlow requires both layer localization and attribute-direction optimization before deployment. The low inference overhead is well supported, but reporting these preparation costs would give a more complete picture of the method’s practical efficiency.
Questions for authors
  • Figures 4 and 7 show that visual interventions change text representations. Do these changes in text states subsequently contribute to demographic bias in the generated images? Clarifying this connection would help support the proposed bias reinforcement loop.
  • With only six images per occupation, the fairness scores may be sensitive to sampling seeds, particularly in the six-group gender×race setting. Please provide uncertainty estimates or results with more samples to assess whether the main comparisons remain consistent.
  • How were the intervention strengths in Tables 1 and 2 selected for each method? Appendix K provides useful strength sweeps, but the criterion for choosing the settings in the main tables is unclear.
  • The low inference overhead in Table 3 is encouraging. It would also be useful to report the time required for layer localization and attribute-direction optimization on each backbone, since these steps are needed before deployment.
6 comments
AuthorsOpenReview ↗

Response to Reviewer tCbT (1)

Thank you for the time and effort you devoted to reviewing our paper. We appreciate your positive assessment of our strong empirical performance, comprehensive ablation studies, and low inference overhead. We address your comments below.

W1 / Q1: Effects of changes in text states on generated images

Thank you for this insightful comment. To examine whether visual-to-text feedback has downstream effects on generated demographic attributes, we added a text-state transfer experiment that provides intervention-based evidence for the visual-to-text-to-visual causal path.

Specifically, we transfer the text states induced by a visual intervention into a separate neutral generation and compare four trajectories:

TrajectoryInput and intervention
(1) SourceNeutral prompt, without intervention
(2) TargetPrompt explicitly specifying an anti-bias attribute, without intervention
(3) Visual patchNeutral prompt, with visual residuals from the Target trajectory injected at the control layers
(4) Text patchNeutral prompt, with text states replaced by those from the Visual patch trajectory

In SD3, transferring the conditional-branch text states during early denoising shifts the generated images toward the target demographic attributes in the anti-bias reference images (Appendix E, Figure 14).

Crucially, trajectory (4) retains the original neutral input text embeddings and pooled conditioning and receives no direct intervention on its image states. Its donor text states come from the visually patched neutral trajectory, rather than the trajectory conditioned on an explicit anti-bias prompt. The resulting attribute changes therefore show that changes in text states induced by visual feedback are sufficient to influence demographic attributes in the final images.

This finding connects the original text-state analyses to observable output changes. Figures 4 and 7 show that visual states shift subsequent text states toward bias or anti-bias reference directions. The new experiment establishes that these shifts can carry attribute information back into image generation. Taken together, these results support the proposed reinforcement mechanism: bias-oriented text states need not merely reflect the evolving visual states, but can also influence subsequent visual generation. Conversely, redirecting this feedback toward anti-bias states can shift the generated attributes accordingly.

We have incorporated this experiment and its implications for the proposed reinforcement mechanism into the revised manuscript (Section 4.3 and Appendix E).

AuthorsOpenReview ↗

Response to Reviewer tCbT (2)

W2 / Q2: Increasing the number of images per prompt for more reliable fairness evaluation

We expanded the main experiments (Tables 1 and 2) to 12 images per prompt using different random seeds and report mean ± standard deviation. The expanded evaluation shows broadly the same trends as the original results.

We completed the expanded evaluation for Vanilla, FairDiffusion, FairImagen, and FairFlow. Due to time constraints, we will provide the expanded results for the remaining baselines no later than the camera-ready version. With 12 seeds, FairFlow achieves the highest fairness in all three settings on FLUX.1 and maintains a strong fairness–fidelity tradeoff on SD3. Across methods, the relatively large fairness standard deviations reflect differences across occupation prompts in initial stereotype strength and debiasing difficulty.

SD3:

MethodGender: Fairness ↑Gender: CLIP ↑Gender: MUSIQ ↑Race: Fairness ↑Race: CLIP ↑Race: MUSIQ ↑Gender × Race: Fairness ↑Gender × Race: CLIP ↑Gender × Race: MUSIQ ↑
Vanilla0.150 ± 0.2480.710 ± 0.0490.744 ± 0.0200.168 ± 0.2010.710 ± 0.0490.744 ± 0.0200.161 ± 0.1390.710 ± 0.0490.744 ± 0.020
FairDiffusion0.655 ± 0.2350.577 ± 0.0710.535 ± 0.1280.487 ± 0.1150.572 ± 0.0730.531 ± 0.1290.503 ± 0.1110.553 ± 0.0690.559 ± 0.121
FairImagen0.192 ± 0.2990.684 ± 0.0610.716 ± 0.0470.221 ± 0.1920.687 ± 0.0580.714 ± 0.0470.179 ± 0.1400.685 ± 0.0620.714 ± 0.048
FairFlow0.636 ± 0.4120.701 ± 0.0490.738 ± 0.0190.738 ± 0.2020.703 ± 0.0480.725 ± 0.0210.503 ± 0.1880.680 ± 0.0420.705 ± 0.019

FLUX.1:

MethodGender: Fairness ↑Gender: CLIP ↑Gender: MUSIQ ↑Race: Fairness ↑Race: CLIP ↑Race: MUSIQ ↑Gender × Race: Fairness ↑Gender × Race: CLIP ↑Gender × Race: MUSIQ ↑
Vanilla0.155 ± 0.2610.710 ± 0.0600.737 ± 0.0150.202 ± 0.2260.710 ± 0.0600.737 ± 0.0150.179 ± 0.1420.710 ± 0.0600.737 ± 0.015
FairDiffusion0.237 ± 0.2940.684 ± 0.0610.738 ± 0.0230.456 ± 0.1290.682 ± 0.0610.741 ± 0.0230.322 ± 0.1290.672 ± 0.0600.741 ± 0.023
FairImagen0.236 ± 0.3210.707 ± 0.0680.731 ± 0.0280.194 ± 0.1860.705 ± 0.0690.731 ± 0.0280.201 ± 0.1550.706 ± 0.0680.731 ± 0.028
FairFlow0.589 ± 0.4010.704 ± 0.0570.730 ± 0.0120.752 ± 0.2060.699 ± 0.0580.731 ± 0.0120.578 ± 0.1460.687 ± 0.0540.717 ± 0.012

W2 / Q3: Selection of intervention strengths for the main experiments

We use the development set of 30 occupation prompts described in Section 6.1 for layer localization, attribute-direction optimization, and intervention-strength selection, and conduct final evaluation on a separate set of 100 non-overlapping occupation prompts. For each model and attribute setting, we sweep intervention strengths on a linear grid using the development set and select a configuration that balances fairness improvement with fidelity preservation. All occupation prompts within the same model and attribute setting use the same intervention strength. The final configurations are listed in Appendix A and remain fixed for the reported test results. We have clarified this selection procedure in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer tCbT (3)

W3 / Q4: Offline costs of layer localization and direction optimization

Using eight NVIDIA RTX 4090 GPUs and the settings in the original manuscript, we measured the following offline runtimes for SD3:

Offline procedureSD3
Layer localization: scanning all layers for one prompt pair143s, approximately 2.38min
Single-attribute direction optimization: per epoch95s on average
Single-attribute direction optimization: full run100 epochs, approximately 2h 39min

The 143-second localization cost is for scanning all layers with a single prompt pair, which can be used to quickly identify candidate control layers. To assess how consistently these layers are identified across prompts and seeds, Figure 9 in Appendix D reports localization results with confidence intervals across 30 prompt pairs and six random seeds for gender and race.

Layer localization is a one-time cost per model, and the identified control layers can be reused for subsequent generations. Once optimized, an attribute direction can be reused for both single-attribute and intersectional debiasing and transferred to out-of-distribution complex scene prompts. Compared with fine-tuning-based methods, storing attribute directions incurs negligible storage overhead. Adjusting intervention strengths or target group ratios requires no additional offline optimization.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the PC, SAC, and AC for their time and effort in overseeing the review process, and all reviewers for their careful reviews and constructive feedback. We appreciate the reviewers’ recognition of our strong empirical performance and low inference overhead (R#tCbT), systematic layer localization and attribute-direction construction (R#ZXvA), mechanistic analysis and simple, reproducible method design (R#mjPD), and comprehensive ablation studies (R#tCbT, R#ZXvA). Below, we summarize the main manuscript revisions and their corresponding reviewer comments. Detailed analyses and point-by-point responses are provided in the response box for each reviewer and the corresponding sections of the revised manuscript.

  • Connecting text-state changes to generated attributes (R#tCbT W1/Q1). Text-state transfer experiments provide causal evidence that visually induced text-state changes affect generated demographic attributes (Appendix E, Fig. 14).
  • Attribute-level validation of control layers (R#ZXvA W1/Q1). DeepFace-based evaluation and visual inspection support LPIPS as a more suitable criterion for control-layer selection in our main experiments (Sec. 4.1; Figs. 8–10).
  • Cross-modal feedback in FLUX.1 (R#ZXvA Q2). Additional experiments show cross-modal feedback consistent with SD3 and demonstrate that FairFlow redirects text states toward anti-bias references (Appendix E, Figs. 11–12).
  • Expanded fairness evaluation (R#tCbT W2/Q2). Expanded comparisons using 12 images per prompt and reporting mean ± standard deviation preserve the original evaluation trends.
  • Intervention-strength selection and offline costs (R#tCbT W2/Q3, W3/Q4). We clarify development-set strength selection and report offline runtimes for layer localization and direction optimization (Sec. 6.1; Appendix A).
  • Discussion of and comparisons with FairGen, DeCoDi, and BADGE (R#mjPD W2). We clarify methodological differences and add SD3 comparisons showing the highest overall fairness and fairness–fidelity tradeoff scores for FairFlow (Sec. 2.1; Appendix C).
  • Challenging settings and image fidelity (R#ZXvA W2; R#mjPD W1). Additional analyses demonstrate strong fidelity preservation while identifying limitations in specific settings and discussing their possible causes (Appendix O; Figs. 1, 16–17, 19).
  • Non-uniform target group ratios (R#ZXvA W3). Five additional target distributions on both models demonstrate adjustable demographic control and improved target-attribute hit rates.
  • Broader demographic coverage (R#mjPD Q1/Q2). Gender × Race × Age experiments and six-category race evaluation demonstrate applicability to more complex attribute combinations and additional demographic categories.

We sincerely welcome further feedback and discussion.

Best regards, Submission 481 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.