- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes a new method for predicting the effect of applying semantic steering to mitigate stereotypes in text-to-image models. This prediction can be made before final image generation, allowing users to apply semantic steering efficiently only to prompts for which the intervention is effective. The paper introduces a metric called Directional Gain (G). G measures the alignment between two directions: the change in the denoiser output induced by steering and the change induced by replacing a stereotypical prompt with its counterfactual counterpart. This allows the effectiveness of steering to be estimated without decoding or evaluating the generated images. Experiments on SD3.5, FLUX, and Qwen-Image show that G can predict the trade-off score between prompt fidelity and stereotype reduction better than the magnitude of the change in the model output.
Critical strengths
- Design of an efficient metric for estimating steering effectiveness. The proposed Directional Gain does not require final image decoding, so the effect of the semantic intervention can be efficiently estimated without waiting for the slow image generation process in diffusion models.
- Evaluation across multiple models and settings. The authors evaluate the proposed method on three different text-to-image models (SD3.5, FLUX, and Qwen-Image), using 1,295 test prompts per model. They also consider two independent image judges, different generation seeds, and varying numbers of denoiser probes. These experiments provide useful evidence of the consistency and computational efficiency of the proposed metric.
Critical weaknesses
-
Insufficient motivation, practicality, and generality of the problem setting. This work focuses on the different effects of semantic steering in diffusion models and aims to predict the effect of an intervention without evaluating generated images. However, I am deeply concerned that the authors do not sufficiently explain when this prediction is necessary or how it can contribute to reducing bias. In particular, the proposed method recommends applying steering only to prompts with high prediction scores and generating images from the original prompts for those with low scores. Although avoiding ineffective interventions can be meaningful in terms of computational efficiency, I still wonder whether it is desirable to generate biased images from unmodified prompts for unselected cases, given the original goal of reducing stereotypes. For example, SAFREE (Yoon et al., 2025) and Concept Removal Guidance (Choi et al., 2026) can adaptively control the range and strength of interventions based on information extracted from semantic representations and denoiser predictions. Based on these prior works, it is unclear whether merely predicting if an intervention is necessary offers advantages over methods that adaptively adjust interventions.
Furthermore, I am concerned about the practicality and generality of the proposed method. In this work, stereotype–counterfactual pairs are needed for both semantic steering and the computation of Directional Gain. It is not clearly stated how such pairs should be prepared for real-world text-to-image generation, making the method dependent on a strong assumption.
Moreover, the authors use a steering method they designed, in which the same prompt pairs are used both to construct the intervention direction and to compute the reference direction for G. Consequently, it remains unclear whether the proposed metric generalizes to other existing steering methods or whether its effectiveness is specific to the intervention setup used in this paper.
-
Lack of comparison with baselines. This work justifies the validity of G by comparing it with the magnitude of the denoiser response and the text-only predictor. I do not think comparisons with these baselines are sufficient to establish the necessity and advantages of the proposed metric.
First, M considers only the magnitude of the change in the denoiser output and does not incorporate any information about the reference direction. Therefore, while comparing G with M is reasonable as an ablation, it is insufficient to demonstrate the superiority of the proposed metric. For example, Concept Removal Guidance (Choi et al., 2026) introduces a metric that estimates the presence of unwanted concepts based on the relationship between the denoiser's noise predictions and the guidance direction. Similarly, SAFREE (Yoon et al., 2025) uses projections onto concept subspaces and cosine similarity in text representations to adaptively adjust interventions. These prior works suggest that stronger alternative metrics leveraging the geometric properties of denoiser predictions and text representations should be considered.
Second, the text-only predictor relies solely on numerical features, such as the norms and distances of text representations, without sufficiently considering the semantic content of the prompt pairs. Figure 4 shows that high-G examples often involve visually explicit attribute changes, such as gender swaps, whereas low-G examples tend to involve changes in personal traits or contextual information. Such differences might be predictable to some extent simply by analyzing the semantic content of the prompt pairs using an LLM. Furthermore, an LLM-based predictor would not require any denoiser evaluations from the image generation model and could be evaluated in advance or asynchronously, independently of the image generation process. These practical advantages make comparisons with stronger LLM-based semantic baselines particularly important. In particular, if G can identify cases in which an intervention appears promising based on textual information but fails to produce the intended effect in practice, such evidence would help establish the value of leveraging model-specific denoiser responses that cannot be inferred from textual semantics alone.
Finally, M and G are compared using the same number of probes despite having different computational costs. M requires only the denoiser outputs before and after steering, whereas G additionally requires denoiser evaluations for the stereotype–counterfactual prompt pair. Thus, M has a lower computational cost per probe. Under the same computational budget, M could potentially use more denoising steps or random seeds. Therefore, a comparison under the same computational budget is necessary.
Choi et al. (2026). Concept Removal Guidance: Evidence-Calibrated Negative Guidance for Safe Diffusion Sampling. ICML 2026.
Yoon et al. (2025). SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image and Video Generation. ICLR 2025.
Questions for authors
Please refer to Weaknesses 1 and 2 above for my main questions and concerns.