- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces TIPGuard for text-to-image prompt moderation. It develops a safety policy framework comprising 528 rules and approximately 2.36 million training examples, uses supervised fine-tuning (SFT) to train moderation models supporting custom policies and fixed categories, and provides fine-grained evaluation datasets.
Critical strengths
- The problem has practical value, with attention to visual scene context and changes in safety policies.
- The data construction process is fairly systematic, combining automated synthesis, hierarchical annotation, and human feedback.
- Defining both prohibited and permitted cases helps clarify safety boundaries and reduce false positives.
- The method performs well on the authors' own benchmarks, while external evaluations also suggest some potential for practical applications.
Critical weaknesses
- Since the user prompts are generated by language models, how is prompt diversity ensured across the two million examples? Could the diversity of the dataset be quantified?
- Why are the baselines in Table 2 not included in the comparisons in Table 3? ImageGuard is an output-side detector, whereas TIPGuard is an input-side detector. Is this comparison fair?
- The evaluation of generalization is insufficient. Can the model understand policies that are explicitly held out from training?
- For the same user input, can the model correctly adjust its judgment when the applicable policy changes?
Questions for authors
See weaknesses.
2 comments
Thanks! My concerns are addressed and I will raise my score.
Weakness 1 - Weakness 4
W1. Dataset diversity and quantitative evaluation
Thank you for suggesting a quantitative assessment of dataset diversity. Our diversity and scalability come from Concept Extraction → Visual Grounding → Keyword-to-Prompt Generation. Each concept expands into visual scenarios: the harmful branch varies subjects, actions, appearance, and risk-related context, while the safe branch varies subjects, settings, and visual styles. Scenario keywords are then converted into complete prompts.
Exact-text deduplication across the Visual Grounding outputs yields:
We further quantify lexical diversity using Distinct-2 (Li et al., 2016), comparing the TIPGuard-2M and T2ISafety datasets. With identical tokenization and a fixed 5,000-token budget, we sample 900 candidate prompts per dataset per run across five different random seeds, reporting the mean and standard deviation.
Under the same token budget, TIPGuard-2M achieves a 10.26-percentage-point higher Distinct-2 score. Therefore, the staged construction process and the quantitative Distinct-2 metric jointly support the diversity of the dataset.
W2. Evaluation protocols and input-side versus output-side comparison
The baselines in Table 2 evaluate prompt moderation before generation, whereas ImageGuard's original experiment in Table 3 evaluates generated images. This work focuses on input-side guards and does not include image generation, so we did not extend the Table 2 baselines to output-side evaluation on generated images.
Regarding the evaluation protocol and comparability, both methods start from the same input prompts and use the same F1 calculation method, making their results comparable. TIPGuard uses the official prompts and their original category labels, with the original policy definitions supplied as inference input and no additional training; ImageGuard evaluates images generated from the corresponding prompts, and its scores are taken from the original paper. The two methods identify risks at the input and output stages, respectively, and this comparison shows that a substantial portion of risks can be identified before image generation.
W3–W4. Generalization to unseen policies and adaptation to policy changes
Apart from TIPGuardBench, all other evaluation benchmarks we use adopt safety policies not seen during training, testing the model's understanding of and generalization to external policies. An unseen policy refers to specific policy definitions not used during training, rather than requiring entirely different risk-category names. Different safety policies can share broad risk categories while differing in their scope, exceptions, and criteria for permitting or blocking a request. At evaluation, external policies are supplied as input to TIPGuard without additional training. The evaluation results in the paper support the model's generalization and adaptation across different external benchmarks and safety policies.
Yes, TIPGuard can adjust its judgments for the same user input according to changes in the applicable safety policy. We added a category-removal experiment on all unsafe TIPGuardBench samples, removing each sample's original top-level category while retaining all others and their definitions. Prompts and model parameters remained fixed. We evaluated safety decisions and category assignments, with 50 randomly selected samples reannotated by humans under their modified policies.
Typical category transitions observed in the experiment include Hate ↔ Discrimination, Suicide & Self-Harm → Violence & Gore, Illegal Activities & Crime ↔ Dangerous Goods, and Sexual Content [Sexual Violence] → Violence & Gore.
Correct policy adaptation means reassessing requests under the remaining policy, rather than allowing every request after its original category is removed. A request can be allowed when its sole applicable restriction is removed; if it still involves another restricted risk, it should be assigned to a remaining applicable category. These results show that TIPGuard can adjust its judgment of the same input in response to policy changes.