- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Paper proposes Safend which insets one fixed refusal demonstration after the adversarial context before the final query. Paper analyses the intervention through contextual contributions and reports substantial reductions in ASR across several models and attacks, supported by utility evaluations and position and content ablations.
Critical strengths
The intervention requires no parameter updates or additional model calls. The experiments report substantial ASR reductions with limited utility degradation.
Critical weaknesses
Safend’s core idea was already proposed in prior In-Context Defence (ICD) work, which suggested placing refusal demonstrations after existing conversations [1]. Using one fixed example instantiates that proposal without establishing a distinct defence mechanism. Furthermore, the original Many Shot Jailbreak paper demonstrated effective mitigation through Cautionary Warning Defense (CWD) [2] as well. The contribution therefore rests primarily on empirical validation, yet the comparison with the closest prior defences remains incomplete.
[1] Zeming Wei et al. (2023). Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations . Section 5.3
[2] Cem Anil et al. (2024). Many-shot Jailbreaking . Section 5.4 and Appendix K
Questions for authors
-
The abstract’s claim that code is available is not substantiated: the linked repository contains only README.md and .gitignore, with no executable implementation.
-
Table 4 describes Goal Prioritization as generating and ranking five candidate responses. This does not match the original method’s setting without training, which prioritizes safety through instructions and in-context examples. Is this a description error or a modification to the baseline? Which implementation was used for the ASR and runtime comparisons?
-
The runtime comparison in Table 4 is insufficiently documented to substantiate the claimed efficiency advantage. Essential measurement details and controls for output length are missing, making the reported overheads difficult to reproduce or compare fairly.