- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a training-free prompting and retrieval pipeline that helps vision-language models moderate images under different safety policies. It also introduces a dataset with detailed content annotations that can be combined into policy-specific safety labels. The authors evaluate their pipeline against existing moderation methods on this benchmark and two external datasets, showing that many existing systems struggle with the proposed benchmark.
Critical strengths
- A training-free method: The pipeline adapts existing VLMs without task-specific fine-tuning, unlike several competing approaches.
- A new, albeit small, benchmark: Rich content annotations support evaluation under multiple moderation policies.
- Competitive empirical results: Strong performance on the proposed benchmark and competitive results on two external datasets, with comparisons against a broad range of methods.
Critical weaknesses
- Same backbone comparisons cover only the extremist ideology subset. I can't find an explanation as to why this particular subset was chosen instead of the entire dataset. Without direct Qwen results on politics, violence, USD, and UnsafeBench, it is difficult to determine how broadly the pipeline improves over its underlying model.
- Component ablations cover only extremist imagery under the strict policy. They do not isolate the contribution of retrieval from structured image inspection and an additional model call
- Annotation reporting is insufficiently detailed. The paper doesn't clarify whether the two annotators were authors or independent annotators, state their qualifications, or report agreement before consensus resolution.
- The dataset is small and narrowly scoped, with images collected from existing benchmarks, AI generation, and web retrieval. Its main added value appears to lie in the policy-reconfigurable annotations rather than the image collection itself,
More minor comments:
- Text within Figures 1 and 3 is a bit too small to read comfortably.limiting its significance as a new data resource.
- The paper SafetyPairs merits discussion as complementary related work on fine-grained image safety.
Questions for authors
- Does the pipeline improve over direct inference with the same Qwen backbone on the political and violence subsets, USD, and UnsafeBench? What motivated restricting the reported same-backbone comparisons to extremist imagery?
- Which baselines support adapting to different moderation policies, and how were the target policies supplied to each? For fixed-policy systems, could performance differences reflect different definitions of harm rather than failures of image understanding. This could be clarified a bit. For example, flagging Nazi imagery even when it explicitly expresses opposition to Nazism?
- Were the annotators authors or independent annotators? Please describe who they are generally and the number of images excluded due to unresolved disagreements.