- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines harmful content within evasive expressions and proposes PASP to improve LLM robustness. PASP combines (i) multi-perceptual inputs that expose textual, phonetic, and visual cues, and (ii) adversarial self-play, where an attacker iteratively generates intent-preserving evasions based on defender feedback while the defender is optimized with an intent-invariance objective. Experiments on ChineseHarm-Bench, EvoHarmBench, and in-the-wild e-commerce data show substantial improvements in robustness against adaptive attacks while largely preserving general model capabilities
Critical strengths
-
The motivation and method are clear, and the visualization is helpful.
-
The authors provide sufficient evidence to support the effectiveness of their proposed method.
-
This paper is well-written, making it easy to follow.
Critical weaknesses
-
Incremental methodological novelty. The two major components of PASP—perception-aware cues (e.g., phonetic/glyph information) and adversarial self-play—are closely related to existing work. The paper should more clearly articulate the technical novelty beyond combining these ingredients with an intent-invariance reward.
-
Limited baselines and lack of frontier-model comparison. Strong contemporary safety models such as WildGuard, LlamaGuard, ShieldGemma, Qwen3Guard, or Shieldstral are not evaluated [1-5]. Moreover, zero-shot evaluation with strong frontier general-purpose LLMs (e.g., GPT-6 Astra) would provide an important reference for assessing whether PASP offers benefits beyond task-specific safety fine-tuning.
-
Limited evidence of in-domain generalization. The experiments mainly rely on within-benchmark train/test splits, while the gains on the in-the-wild data are considerably more modest. It therefore remains unclear whether PASP generalizes beyond the benchmark distributions. Stronger cross-dataset evaluation (e.g., training on ChineseHarm-Bench and testing on EvoHarmBench), as well as evaluation on English harmful-content detection ( represents a fundamental setting for general-purpose content moderation) would better establish its practical generalizability.
-
Lack of cross-lingual OOD generalization. Beyond in-domain generalization, it remains unclear whether PASP transfers to substantially different linguistic systems. Since the current implementation relies on Chinese-specific cues such as Pinyin and character-level obfuscation, evaluation on OOD languages with different writing and phonological systems, such as Korean or Arabic, would help determine whether PASP learns a language-agnostic perception-aware mechanism or primarily Chinese-specific robustness.
[1] Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms
[2] Llama guard: Llm-based input-output safeguard for human-ai conversations
[3] Shieldgemma: Generative ai content moderation based on gemma
[4] Qwen3guard technical report
[5] Shieldstral
Questions for authors
Refer to weaknesses.