- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
In this work, the authors present the first mm knowledge graph (mmkg) of unsafe concepts/relations, followed by an SSM, or semantic safety margin (a non-parametric score). This serves as a similarity constraint over the mmkg that specifically measures the activation of harmful versus safe concepts. They propose TIDIER, which provides alignment in both prompt and trajectory spaces via cross-attention intervention. It is tested across 4 benchmarks.
Critical strengths
Mapping harmful phrases to explicit and implicit categories (trope patterns) is an important distinction. The expansion of the harm concept is a theoretically important idea to keep the evolution of harm terms covered.
Critical weaknesses
- The explanations are very unclear and difficult to follow, making it hard to understand what is novel or interesting, or simply how the methods are implemented. Ultimately, the presentation of this work is its most crucial flaw. Because the explanations and methodologies are so unclear, the work cannot be assessed properly. While the questions raised here and below are important, the most significant gap in this paper is the lack of a cohesive story, tightness, and overall clarity. Addressing these presentation issues is essential before the technical contributions can be fairly evaluated.
Section 3: The process of structuring the mmkg graph is very unclear, with an increased usage of unfamiliar terms lacking proper explanation.
Section 4: Are the embeddings during diffusion the average over the denoised image patches? What constitutes a harmful activation, and how was the threshold chosen?
Section 5: What does it mean that the terms "are checked only when harm can enter" (line 207)? Is the metric not applied continuously to detect this automatically? Furthermore, what exactly is the cross-attention "selected activation" intervention?
- Weak results:
Regarding Table 1, there is no proper explanation of the Toxicity and CMMD metrics (such as their scale or how to interpret them). The types of baselines used are also unclear, aside from the fact that they are inference-time methods. What about other common attention interventions or steering methods? Additionally, the changes in the CLIP score are minor (~0.01-0.03), indicating very little improvement in preservation.
Qualitative results: There is a lack of qualitative examples comparing generated images across baselines, as well as for the repaired prompts. The three qualitative examples provided in the appendix are not fully aligned with the prompt's "neutral" intention.
Missing ablations: Ablation studies on the method are missing for the prompt and trajectory, the fused verifier, the diffusion timestamps, the rewind repair step, the fidelity budget, and global versus local interventions. While some of these are presented in Tables 4 and 5, they are very difficult to understand. Finally, the CLIP image representation is weak; have other encodings been tried (e.g., DINOv3)?
Questions for authors
Interesting and missing statistics: Are there statistics showing when an unsafe prompt is a combination of benign concepts versus when it is directly unsafe within relevant corpora or real-world prompt usage? Additionally, are there statistics for unsafe or harmful concepts categorized across different domains (e.g., cultural, sexual)?
Section 6: Why is an external harmfulness classifier used, and why was that specific one chosen?
Prompt-space evaluation: What prompt was used for the LLM-based prompt rewriter? Was it required to maintain as much of the original prompt phrasing as possible and only drop or change the harmful parts?
Figure 5: It seems that the harmful elements are separable from the safe ones in earlier steps (around step 20). Why, then, is it considered separable enough only at step 46? Furthermore, this implies that most of the harm is in the fine details rather than being structural, as the image structure is established in the early steps. Do you have any statistics regarding the visual nature of the harm (e.g., resolution, size, attributes, or shape)?
Conclusions: You mention that the method is an "effective alternative to prompt filtering and model editing." Is it effective in terms of computational performance and inference time? This aspect is completely missing here.
MMKG prompt robustness: What happens when there are multiple options in the MMKG, where one indicator can lead to many concepts? How will the prompt phase-match step act in this scenario? Additionally, how robust is the method to different phrasings of the same prompt?
Writing and Presentation: Abstract: The sentence regarding OWCI (lines 21-22) seems detached and unrelated. Introduction: The sentence spanning lines 45-47 is very long and unclear. The SSM equation appears twice (in Section 4 and again in Section 5.2). FAQ section: The motivation for many of the design choices should be clarified within the main paper rather than placed in a separate FAQ section before the appendix. Consider making this information more concise and integrating it directly into the main text.