- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how prompt modifiers, such as artist names, styles or mood words, affect social risks in t2i models. The authors build PMD dataset of real prompts with labeled modifiers and try to identify which modifier types change the output. They then test those modifier types for effects on gender and race representation, how easily fake-image detectors flag the images, and on unsafe content.
Critical strengths
- An interesting perspective on how the prompt modifiers, such as artist names, styles or mood words, affect social risks in t2i models.
- The released dataset of annotated real-world prompts should be useful to the community.
- The evaluation covers diverse dimensions, such as demographic bias, fake-image detectability, and content safety, providing a broader view of how modifier effects.
Critical weaknesses
-
Novelty relative to prior work. [1] already show that modifiers can unlock unsafe generation, and [2] study modifiers and bias. The added contribution seems to be the PMD dataset and the category-level organization.
-
FairFace classifiers and the HuggingFace deepfake detector are trained on photographic faces or images. The independent variable here is style. Classifier errors are therefore likely to correlate with the modifiers in the study.
-
Safety spans come from MMD, e.g., "sinister", "horror art", "Beksiński", "medical illustration" (Fig. 12). It is not surprising that "sinister" raises Q16 gore flags on "a person." Thus the safety experiment may be confounded by how spans were chosen.
[1] Modifier Unlocked: Jailbreaking Text-to-Image Models Through Prompts (2025)
[2] Can Prompt Modifiers Control Bias? A Comparative Analysis of Text-to-Image Generative Models (2024)
Questions for authors
- What is the main contribution beyond [1, 2] ? (weakness1)
- How modifiers affect the classifier? (Weakness2)
3 comments
Response to Reviewer 4jMn (Part 2)
W2, Q2. Classifier errors and the effect of the modifiers on classifiers.
We agree that classifier errors may correlate with style. We therefore measure how modifiers affect each classifier directly.
FairFace classifiers. We validate them against human labels on a stratified sample of 460 images: the 100 images in Section 5 plus 360 new images (20 per model × group), covering the baseline and all five steering categories on the three models. Two authors independently label perceived gender and race. Agreement is 96.7% for gender (κ = 0.94) and 92.0% for race (κ = 0.82), and disagreements are resolved by discussion. The table gives the accuracy of each classifier against the consensus labels.
| Classifier | Baseline | Artist | Atmosphere | Medium | Movement | Trending |
|---|---|---|---|---|---|---|
| Gender (FairFace Gender Image Detection) | 0.95 | 0.92 | 0.91 | 0.97 | 0.88 | 0.93 |
| Race (fairface-race-vit) | 0.92 | 0.82 | 0.91 | 0.84 | 0.86 | 0.92 |
- Gender. Accuracy stays between 0.88 and 0.97 in every category, close to the baseline (0.95).
- Race. Accuracy is 0.92 at baseline and 0.82–0.92 with modifiers. Modifiers lower it by at most 0.10, and every category stays above 0.80.
- Classifier errors cannot explain the shifts. The shifts in Section 4.2 are much larger than these error rates. For example, "a CEO, alphonse mucha" on FLUX.1-dev turns 100% men into 100% women, while at most 8% of artist images are misclassified for gender. Modifiers therefore affect the FairFace classifiers only moderately. They are reliable for gender and race.
Deepfake detector. We also agree with the reviewer that the original detector is trained on face images. In our revised version, we apply it to person images only, which lie within its training domain. We further add a second detector, Organika SDXL-detector, for both person and street images. It is fine-tuned from an AI-art detector trained on images in many artistic styles, and its real images cover general scenes rather than faces only. To see how modifiers affect detection, we label each image as photorealistic or stylized with CLIP zero-shot classification (88% agreement with human labels on 156 images). We then fit a logistic regression of detection on category and model, with and without a photorealism indicator.
- Modifiers mainly change style. Except for atmosphere, they lower the photorealistic share of portraits from 79% to 5–9%.
- The original detector responds to this style change. All five categories significantly lower its detection of portraits, but none remains significant once photorealism is controlled.
- Category effects remain under the second detector. After controlling for photorealism, artist and medium still lower the detection of portraits (coefficient −1.19, p = 0.004), and atmosphere, movement and trending still raise the detection of street images (+0.58 to +0.63, all p < 0.05).
Modifiers therefore affect the original detector mainly through style. The second detector covers stylized images, and its category effects remain after controlling for style. The category-level effects on detectability are therefore not explained by detector failure on stylized images.
We will report the 460-image validation with the per-category table, and add the second detector and the photorealism analysis to our revised paper.
Response to Reviewer 4jMn (Part 3)
W3. Choice of safety spans.
Our goal in the safety experiment is to compare how modifier categories affect unsafe generation. To measure this effect, we deliberately choose modifiers that can trigger unsafe outputs. This is because common modifiers rarely trigger them. With the five most frequent spans of each category in PMD, the gore rate is 10% (13% at baseline) and the explicit rate is 1.5% (0% at baseline). These spans can cause a small amount of unsafe generation, but they cannot give a convincing comparison between categories.
We therefore use unsafe-associated spans from MMD. This is not jailbreak prompt design. We do not craft or optimize prompts. We only group the released spans by category, and the spans of every category are chosen in the same way. We agree that a span such as "sinister" can raise gore flags by itself. However, this does not explain the differences between categories, which are the focus of our analysis.
Gore rate pooled over the three models:
| Category | Frequent spans (PMD) | Unsafe-associated spans (MMD) |
|---|---|---|
| Baseline | 12.7% | – |
| Artist | 8.7% | 42.0% |
| Atmosphere | 10.7% | 95.3% |
| Medium | 10.0% | 30.5% |
| Movement | 6.7% | 74.7% |
| Trending | 14.0% | 14.7% |
- The same selection gives very different outcomes. Under unsafe-associated spans, the gore rate ranges from 15% (trending) to 95% (atmosphere). "Beksinski" (artist) and "medical illustration" (medium) are also unsafe-associated, yet these two categories produce much less gore (42% and 31%). If the span selection drove the result, all categories would rise in a similar way.
- Category effects remain after controlling for span wording. We score each span by its CLIP text similarity to "gore, violence, blood" and add this score to a logistic regression of gore on category and model (artist as reference). Atmosphere remains significantly higher (coefficient +1.09, p < 0.001), and medium and trending remain significantly lower (p ≤ 0.008). The span score itself has no significant effect (p = 0.64). For explicit content, the category differences also remain significant after this control.
The category ranking therefore reflects the modifier categories rather than how the spans were chosen. In the revised paper, we will add this analysis.
Response to Reviewer 4jMn (Part 1)
We thank the reviewer for the weak acceptance rating, and for the insightful and constructive comments. We are confident to address all the concerns raised in weaknesses and questions.
W1, Q1. Contribution beyond modifier-based jailbreaking and modifier-controlled bias.
We clarify that the open question ModEval answers is which categories of prompt modifiers steer T2I generation, and which of these categories drive which social risks. Prior work, including [1] and [2], treats modifiers as a single, undifferentiated prompting tool. ModEval instead analyzes them at the
category level.PMDis also a new modifier dataset we constructed for future modifier and T2I model research.Novelty over modifier-based jailbreak attack. [1] shows that modifiers can unlock unsafe generation, but it assumes an attacker who deliberately crafts or searches for modifiers that bypass safety filters. ModEval DOES NOT require an attacker. Even a benign user can generate unsafe images by accident, because many unsafe-associated modifiers are ordinary style words such as "eerie" or "ominous". Our question is therefore not whether a filter can be bypassed. Instead, we ask which steering categories make unsafe outcomes more likely. Beyond safety, we also analyze other social-risk dimensions, which enables a cross-risk analysis.
Novelty over modifier-controlled bias. [2] asks whether prompt modifiers can control bias. It treats modifiers as an intentional tool for adjusting demographic outcomes. ModEval asks the opposite question. We study ordinary style modifiers that users add for aesthetic reasons. These modifiers contain NO demographic words, and we ask whether they still change demographic representation as a side effect. Our bias modifiers are the most frequent spans of each steering category in real-world prompts, rather than spans selected to control bias. Beyond bias, we also analyze other social-risk dimensions, which enables a cross-risk analysis.
[1] Modifier Unlocked: Jailbreaking Text-to-Image Models Through Prompts (Liu et al., IEEE S&P 2025)
[2] Can Prompt Modifiers Control Bias? A Comparative Analysis of Text-to-Image Generative Models (Shin et al., IJCAI Workshop 2024)