- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces UNSAID, a benchmark for evaluating unsafe behavior in T2I models. The benchmark asks whether models generate norm-compliant visual configurations when a benign prompt leaves a safety-relevant detail unspecified. It contains around 500 safety norms and ~1000 prompts, which are evaluated across nine models. The authors also test generic and explicit safety prompting, repeated sampling, existing image guards, and an agentic prompt refinement harness.
Critical strengths
- The paper identifies a concrete form of underspecified prompt behavior and evaluates several recent T2I models.
- They distinguish between between safe, violation, ambiguous, and not depicted is more informative than a binary label.
- The paper evaluates several possible interventions rather than presenting only a benchmark.
Critical weaknesses
- Extensive prior work already audits biases and safety failures in text-to-image models. UNSAID applies this established framework to a narrower set of prescriptive safety norms without adequately discussing close work such as OpenBias.
- The benchmark contains only 498 norms and 1,144 prompts, with roughly two prompts per norm. This restricts linguistic, contextual, cultural, and visual diversity.
- Questionable annotation reliability: Much of the pipeline is AI-assisted, and large-scale evaluation depends on a single VLM judge. Only 400 images are human-labeled, the same sample is used to select and evaluate the judge, and individual votes and disagreement metadata are not reported. This to me appears to be the biggest weakness along with the limited scale of the dataset.
- The paper does not establish a compelling deployment scenario or clearly distinguish depicting unsafe conduct from producing harmful content. The failure of existing guards may reflect that their native policies were not designed to enforce these scene-specific norms.
- Presentation quality: Figures 3, 4, and 5 have extremely small labels that are difficult to read.
Questions for authors
- How do you justify the benchmark’s limited scale, particularly given that the images are AI generated and much of the annotation is automated?
- Why is the same 400 image set used to select and evaluate the automated judge?
- Will you release individual annotator votes and disagreement metadata?
- What practical deployment scenario motivates treating depictions of norm violations as safety failures, and how do you distinguish a guard’s visual-reasoning failure from a mismatch with its native safety policy?
7 comments
Response to Reviewer h6GD (Part 2)
Response to W3, Q2, and Q3
Thank you for raising these concerns. We re-emphasize that UNSAID is constructed primarily by human annotators, with LLMs used solely to draft candidate prompts that subsequently undergo rigorous human verification. We select our judge based on agreement with model-agnostic human gold labels. Thus, performance on this validation set demonstrates that the model aligns closely with human decisions, a methodology well-established in prior work [1–3].
We independently compared three candidate judges and selected Qwen3.7-Plus. To further validate this choice, we annotated a separate set of 900 images (100 per generator) and evaluated all three judges against the human labels:
| Evaluator | ND (F1) | Amb (F1) | Viol (F1) | Safe (F1) | Avg. F1 | |||
|---|---|---|---|---|---|---|---|---|
| Qwen3.7-Plus | .778 | .570 | .881 | .917 | .786 | .781 | .876 | .798 |
| Gemini-3.1-Flash | .810 | .592 | .854 | .890 | .787 | .761 | .832 | .764 |
| GPT-5.6-Terra | .694 | .457 | .818 | .865 | .708 | .681 | .797 | .721 |
| Human -- Human | — | — | — | — | IAA$_4 = .796$ | .697 | .807 | IAA$_2 = .904$ |
Qwen3.7-Plus achieves the highest , and (Spearman’s correlation coefficient), while its average F1 is nearly identical to Gemini's. These results support our judge selection and the reliability of the subsequent evaluation. We also report human inter-annotator agreement and will include the full confusion matrices in the appendix.
Response to W4 and Q4
Thank you for raising these questions. We respectfully disagree that UNSAID lacks a compelling deployment scenario or that the guard failures can be explained solely by differences in native safety policies. The reasons are as follows.
-
Compelling deployment scenarios. UNSAID studies a common deployment condition: users specify the intended activity but omit safety-critical visual details, leaving the model to determine how the scene is depicted. When the model defaults to an unsafe configuration, it can present hazardous conduct as normal or acceptable. This issue is inherently critical because unsafe visual defaults can normalize severe or life-threatening hazards in the real world. Representative applications include:
- Instructional, educational, and child-facing media: Automated illustrations for manuals, health guidance, school materials, and children's content, where unsafe depictions may mislead users or vulnerable groups [4].
- Commercial, editorial, and social-media generation: High-volume production of workplace, product, and lifestyle imagery, where manual inspection of every generated image is impractical [5].
- Multimodal training and embodied planning: Synthetic training data and visual task planning for embodied agents, where unsafe visual defaults can propagate into downstream representations and physical decisions [6].
-
Implicit safety and guard-policy coverage. UNSAID violations differ from conventionally prohibited content (e.g., nudity or hate symbols), they reveal a critical blind spot: benign prompts that are fulfilled literally but depict unsafe configurations. A comprehensive safety guard should recognize these implicit hazards rather than restrict itself to narrow explicit taxonomies. This limitation highlights a vital coverage gap rather than irrelevance.
Furthermore, these failures stem from more than missing native policies. As shown in Table 6, providing explicit safety criteria and rubrics (+ Criterion and + Rubric) fails to substantially improve detection, confirming that current guards lack the underlying physical and spatial reasoning required. We will clarify these deployment scenarios and Table 6 in the revised manuscript.
Response to W5
Thank you for pointing this out. We have adjusted the figures accordingly for better readability and updated them in the revised manuscript.
References
- [1] ACL, 2026, Chen et al., FlexGuard: Fast and Flexible LLM Guardrails via Dynamic Rules
- [2] ICLR, 2026, Shayegani et al., Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directedness
- [3] NAACL, 2025, Mou et al., AgentSense: Benchmarking Social Intelligence
- [4] CVPR, 2024, Liu et al., Intelligent Grimm: Open-ended Visual Storytelling via Latent Diffusion Models
- [5] CCS, 2023, Qu et al., Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models
- [6] ICLR, 2024, Black et al., Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models
Thank you for the detailed response and additional experiments. The independent 900-image validation substantially addresses my concern about judge reliability, and the deployment examples clarify the benchmark’s motivation. I will raise my soundness score from 1 to 2.
My remaining concerns are the limited contextual coverage per norm and the distinction between depicting a norm violation and constituting a safety failure, particularly for social conventions. Please also clarify whether individual annotator votes and disagreement metadata will be released.
I currently retain my weak-reject rating, but am open to acceptance if the discussion establishes a convincing consensus on the benchmark’s contribution and scope.
Response to Reviewer h6GD (Part 1)
We thank the reviewer for their time and effort, and for the constructive feedback. We carefully address the concerns below.
Response to W1
Thank you for pointing out this related work. We will expand Section 2 to include a discussion of OpenBias. However, we respectfully disagree that UNSAID is merely a narrower subset of OpenBias. The novelty and significance of UNSAID lie in uncovering potential safety biases ranging from social discomfort to severe physical harm. This focus is crucial because, unlike general bias, overlooking these issues can cause discomfort and even real-world harm. Specifically:
Response to W2 and Q1
Thank you for raising these questions regarding the scale and diversity of UNSAID. We would first like to clarify two potential misinterpretations:
Furthermore, our benchmark already offers substantial diversity and competitive scale:
Regarding contextual settings, we exclude rules requiring temporal sequences or hidden intent because static T2I generation can be fairly judged only on observable single-frame evidence. This ensures a more robust evaluation. Additionally, we restrict UNSAID to norms with broad, cross-cultural safety consensus, rather than treating culturally contingent customs as universal safety violations, thereby preventing potential normative bias. We will add these discussions to the Limitations and Future Work sections to prevent misunderstandings.