acceptodds
Withdrawn from ICLR 2027

TIDIER: Text-to-Image Alignment via Knowledge-Guided Hate Detection, Interpretation, and Repair

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-Image diffusion models are widely deployed but remain unsafe, generating hateful stereotypes, demographic bias, and dehumanizing compositions under both adversarial and benign prompts. Existing safety methods rely on endpoint filtering or global trajectory steering over unstructured signals, leading to over- censorship, poor generalization, and failure on compositional harms. We instead introduce the Semantic Safety Margin (SSM), a structured similarity constraint over a Multimodal Knowledge Graph (MMKG) encoding harmful concepts, benign counterparts, and contextual relations. Building on this, we propose TIDIER, a training-free framework with dual-space alignment: prompt-space repair and trajectory-space intervention. Open-World Concept Inference (OWCI) detects novel harmful semantics and proposes them for human-reviewed graph expansion. Across four benchmarks (Detonate, I2P, T2I-Safety, T2I-RiskyPrompt), TIDIER reduces prompt-level toxicity by 65% and image toxicity by up to 64% while achieving a 9–10% relative gain in CLIP score. The code and MMKG are available at https://anonymous.4open.science/r/TIDIER-ICLR-148D/ .

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety, Knowledge Graph, Alignment, Diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_z82ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this work, the authors present the first mm knowledge graph (mmkg) of unsafe concepts/relations, followed by an SSM, or semantic safety margin (a non-parametric score). This serves as a similarity constraint over the mmkg that specifically measures the activation of harmful versus safe concepts. They propose TIDIER, which provides alignment in both prompt and trajectory spaces via cross-attention intervention. It is tested across 4 benchmarks.

Critical strengths

Mapping harmful phrases to explicit and implicit categories (trope patterns) is an important distinction. The expansion of the harm concept is a theoretically important idea to keep the evolution of harm terms covered.

Critical weaknesses
  1. The explanations are very unclear and difficult to follow, making it hard to understand what is novel or interesting, or simply how the methods are implemented. Ultimately, the presentation of this work is its most crucial flaw. Because the explanations and methodologies are so unclear, the work cannot be assessed properly. While the questions raised here and below are important, the most significant gap in this paper is the lack of a cohesive story, tightness, and overall clarity. Addressing these presentation issues is essential before the technical contributions can be fairly evaluated.

Section 3: The process of structuring the mmkg graph is very unclear, with an increased usage of unfamiliar terms lacking proper explanation.

Section 4: Are the embeddings during diffusion the average over the denoised image patches? What constitutes a harmful activation, and how was the threshold chosen?

Section 5: What does it mean that the terms "are checked only when harm can enter" (line 207)? Is the metric not applied continuously to detect this automatically? Furthermore, what exactly is the cross-attention "selected activation" intervention?

  1. Weak results:

Regarding Table 1, there is no proper explanation of the Toxicity and CMMD metrics (such as their scale or how to interpret them). The types of baselines used are also unclear, aside from the fact that they are inference-time methods. What about other common attention interventions or steering methods? Additionally, the changes in the CLIP score are minor (~0.01-0.03), indicating very little improvement in preservation.

Qualitative results: There is a lack of qualitative examples comparing generated images across baselines, as well as for the repaired prompts. The three qualitative examples provided in the appendix are not fully aligned with the prompt's "neutral" intention.

Missing ablations: Ablation studies on the method are missing for the prompt and trajectory, the fused verifier, the diffusion timestamps, the rewind repair step, the fidelity budget, and global versus local interventions. While some of these are presented in Tables 4 and 5, they are very difficult to understand. Finally, the CLIP image representation is weak; have other encodings been tried (e.g., DINOv3)?

Questions for authors

Interesting and missing statistics: Are there statistics showing when an unsafe prompt is a combination of benign concepts versus when it is directly unsafe within relevant corpora or real-world prompt usage? Additionally, are there statistics for unsafe or harmful concepts categorized across different domains (e.g., cultural, sexual)?

Section 6: Why is an external harmfulness classifier used, and why was that specific one chosen?

Prompt-space evaluation: What prompt was used for the LLM-based prompt rewriter? Was it required to maintain as much of the original prompt phrasing as possible and only drop or change the harmful parts?

Figure 5: It seems that the harmful elements are separable from the safe ones in earlier steps (around step 20). Why, then, is it considered separable enough only at step 46? Furthermore, this implies that most of the harm is in the fine details rather than being structural, as the image structure is established in the early steps. Do you have any statistics regarding the visual nature of the harm (e.g., resolution, size, attributes, or shape)?

Conclusions: You mention that the method is an "effective alternative to prompt filtering and model editing." Is it effective in terms of computational performance and inference time? This aspect is completely missing here.

MMKG prompt robustness: What happens when there are multiple options in the MMKG, where one indicator can lead to many concepts? How will the prompt phase-match step act in this scenario? Additionally, how robust is the method to different phrasings of the same prompt?

Writing and Presentation: Abstract: The sentence regarding OWCI (lines 21-22) seems detached and unrelated. Introduction: The sentence spanning lines 45-47 is very long and unclear. The SSM equation appears twice (in Section 4 and again in Section 5.2). FAQ section: The motivation for many of the design choices should be clarified within the main paper rather than placed in a separate FAQ section before the appendix. Consider making this information more concise and integrating it directly into the main text.

Reviewer_bkh1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets hateful and stereotyped text-to-image outputs. It argues that a scalar toxicity score or a single unsafe direction cannot represent harm arising from combining two individually benign concepts, and that suppressing such a direction tends to erase identity. TIDIER builds a multimodal knowledge graph of hate concepts, indicator terms, subject–predicate–object trope patterns and neutral counterparts, and claims to enforce one similarity constraint, the Semantic Safety Margin, both in prompt repair and in late-stage detection with trajectory rewinding. OWCI reportedly detects harmful concepts missing from the graph and proposes them for human review. On 1,000 prompts from four benchmarks with SD-v1.5 and SDXL, the paper reports lower toxicity and higher CLIP scores than training-free and model-editing baselines.

Critical strengths

S1. The problem framing is concrete. Modeling a hateful image as a relation between an identity and a predicate or object, with an identity-preserving neutral target for repair, matches how stereotypes are expressed more closely than steering away from a single unsafe direction. It also leaves model weights unchanged.

S2. On SD-v1.5, the overall toxicity reduction holds under three external evaluators with essentially the same method ranking, so the result does not hinge on the primary classifier.

Critical weaknesses

W1. The method as described does not match the central "similarity constraint" claim Sec. 1 states that the contribution is to replace a scalar harm score with a similarity constraint, enforced in both the prompt and the trajectory (Sec. 5). However, none of the three prompt paths in Sec. 5.1 (indicator matching, template matching, embedding-threshold comparison) uses the difference between harmful and safe activation defined in Sec. 4, and the trajectory repair in Sec. 5.2 also fires when the harmful activation alone crosses its threshold. I could not find how the safe-side prototypes νs\nu_s are built. As described, the method consists of matching against graph entries, template substitution, and verifier-triggered rewinding. The margin itself is unused in the prompt paths and is only one of two trigger conditions in the trajectory, and no ablation compares it with the harmful activation alone.

W2. The OWCI statistics in Table 6 are inconsistent with the novelty definition in Sec. 6 In Sec. 6, novelty is one minus the mean similarity to the top-K known concepts, and sknowns_{\text{known}} is the maximum of those similarities. If TopSim in Table 6 is sknowns_{\text{known}}, Novelty cannot be lower than 1 − TopSim, because the mean of the top-K similarities cannot exceed their maximum. A sample unlike every known concept must therefore be novel. Table 6 nevertheless reports TopSim 0.28 and Novelty 0.18 for safe-context prompts, where Novelty should be at least 0.72, and 0.73 and 0.21 for known-harmful prompts. Table 6 is the main evidence that OWCI separates withheld harmful concepts from known and benign inputs, so as reported it cannot support that claim.

Questions for authors
  1. How were the 400 implicit prompts in Table 3 selected, and how were their labels obtained? If the split depends on the method's own matching, the implicit bucket is not independent of Path B, and the gain in implicit F1 from 0.41 to 0.79 is hard to interpret.

  2. Table 1 has only two training-free baselines. Training-free Safe Denoisers (Kim et al., 2025) and PurifyGen (Cao et al., 2025b), cited in Sec. 2, are direct counterparts to the trajectory and prompt branches; why are they not compared? Were the existing baselines configured for hate categories, and how were they run on SDXL, where SLD's toxicity (0.32) is higher than Vanilla's (0.31)?

  3. The paper reports an Id-Pres of 0.89 as 89% of benign prompts left unmodified, and Id-Pres and OC sum to 1.00 in every row of Table 3. Such a metric would count a rewrite that keeps the identity term the same as one that removes it. The LLM baseline (gemma-4-E2B), which the paper says rewrites nearly every prompt, has the lowest toxicity (0.12). How does TIDIER compare with an LLM that rewrites only prompts it first judges harmful, under a metric that checks whether the identity term survives?

  4. Post-Harm in Table 5 is also the score used to trigger repair and to accept iterative updates. Do you have results for the repair variants from an evaluator that does not use the MMKG?

  5. I found no human evaluation of generated images; their safety is judged entirely by classifiers. How well do general-purpose classifiers such as LlavaGuard and Q16 detect compositional stereotypes, for example an identity paired with a benign object?

  6. Which experiments does "adversarial settings" in Sec. 1 refer to? How does TIDIER perform on misspelled or paraphrased prompts, or on prompts searched against this defense, for example with Prompting4Debugging?

  7. Sec. 7 states that CLIP in Table 1 is computed "against the original images", yet TIDIER-PT scores higher than Vanilla (0.340 vs. 0.310 on SDXL). What is the reference for CLIP? Is Tox in Table 3 computed on text or on images?

  8. All 200 benign prompts are identity-sensitive. What are the image quality and the verifier's false-trigger rate on ordinary prompts such as COCO captions, and what reference set is used for CMMD?

  9. Which experiment does the TIDIER point (0.93, 0.89) in Fig. 1 come from? It does not match the 0.91 and 0.81 in Table 3.

Reviewer_T6PHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors address compositional harm in text-to-image (T2I) generation, a problem that remains insufficiently addressed by existing methods. They propose TIDIER, a training-free framework built around a Multimodal Knowledge Graph (MMKG) that represents harmful concepts, benign counterparts, and their relations using text and visual prototypes. The MMKG supports the detection of compositional harm and provides neutral targets for prompt-space repair and trajectory-space intervention. TIDIER also incorporates Open-World Concept Inference (OWCI) to identify novel harmful semantics and propose candidates for human-reviewed graph expansion. The authors evaluate TIDIER through extensive experiments.

Critical strengths
  • The paper addresses the important problem of compositional harm in T2I generation, where harm arises from relations between otherwise benign concepts.
  • The introduced Multimodal Knowledge Graph offers a promising approach to representing concept relations and addressing compositional harm. Open-World Concept Inference (OWCI) further supports future extensions by identifying harmful semantics beyond the current concept set and proposing candidates for human-reviewed inclusion.
Critical weaknesses
  • The writing needs improvement, as the current presentation is confusing and difficult to follow. The paper’s organization could also be improved. Sections 3–6, which describe the proposed method, could be consolidated into fewer sections. Key details, including the definition of compositional harm and the construction of the MMKG, should be explained more clearly in the main paper rather than primarily in the FAQ section or appendices.

  • The baseline comparison omits relevant recent methods, such as CASG [1], CASteer [2], SafetyDPO [3].

  • The experiments are limited to U-Net-based models, namely SD 1.5 and SDXL. It remains unclear how the proposed method would perform on DiT-based models such as SD3 and FLUX.

  • The authors introduce an evaluation setup focused on compositional harm. However, it remains unclear how the proposed method would perform under established T2I safety evaluation protocols used by methods such as SLD and CASG.

  • The authors assemble an evaluation subset containing 400 explicit, 400 implicit/compositional, and 200 benign safe-context prompts. -However, the selection criteria, annotation procedures, and composition of this subset are insufficiently documented.

  • How would performance change if the repaired prompts were used with existing methods such as SLD and SAFREE?

[1] When Safety Collides: Resolving Multi-Category Harmful Conflicts in Text-to-Image Diffusion via Adaptive Safety Guidance. CVPR 2026
[2] CASteer: Cross-Attention Steering for Controllable Concept Erasure. ICLR 2026
[3] AlignGuard: Scalable Safety Alignment for Text-to-Image Generation. ICCV 2025

Questions for authors

Please see the Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.