- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes multi-concept detection, a jailbreak prompt detector. The authors conduct experiments on several models, datasets, and jailbreak attacks.
Critical strengths
- Table 1 covers a wide range of jailbreak attack methods, and detection baselines.
Critical weaknesses
-
The idea is not novel. Jailbreak detector has been a well-studied topic. It has also been works designing jailbreak defenses based on internal activations/features.
-
The defense lacks practical considerations. How much additional time/compute does it require to run the defense during deployment? The largest LLM used in this paper is merely 13B. And the LLMs are generally out-of-dated. In summary, I have serious doubt about the method's performance and effiency on large-scale, latest LLMs such as Qwen3.6-35B.
-
The most concerning shortcoming of the proposed method is the need to enumerate exisitng jailbreak attacks, and the authors did not conduct evaluation on unseen/OOD jailbreak attacks. In practise, the attackers very likely to use attack algorithms never published on papers. As a result, it is important to show that the detector is still effective on such attacks.
Questions for authors
Please see weakness.