acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Many-shot Jailbreak Attacks with One Demonstration

Abstract

Many-shot jailbreaking (MSJ) causes safety-aligned language models to answer harmful queries by conditioning them on many harmful question-answer demonstrations. We study whether this contextual influence can be countered during inference while preserving useful responses. We introduce Safend, an inference-time defense that restores refusal behavior with a single fixed safety demonstration. We characterize MSJ and Safend as competing contextual response patterns. Harmful demonstrations repeatedly associate unsafe requests with compliant responses, while Safend inserts a fixed safety-aligned refusal demonstration after the harmful examples and immediately before the target query. This view treats harmful compliance and refusal as competing response patterns whose influence depends on contextual aggregation rather than example count alone. Safend requires no parameter updates, request-specific optimization, or white-box access, and reuses the same demonstration across inputs. Experiments on open-weight and API-based LLMs show that Safend reduces the average attack success rate of context-based jailbreaks from 76.7% to 2.3% on Llama-3.1-8B-Instruct and from 46.6% to 2.3% on GPT-4o. These reductions are achieved while preserving general capabilities, long-context performance, and benign response behavior. These results establish a simple and practical approach for recovering safety from adversarial contextual adaptation at inference time. Code is available at https://anonymous.4open.science/r/Safend.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety alignment, many-shot jailbreaking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YbEeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Paper proposes Safend which insets one fixed refusal demonstration after the adversarial context before the final query. Paper analyses the intervention through contextual contributions and reports substantial reductions in ASR across several models and attacks, supported by utility evaluations and position and content ablations.

Critical strengths

The intervention requires no parameter updates or additional model calls. The experiments report substantial ASR reductions with limited utility degradation.

Critical weaknesses

Safend’s core idea was already proposed in prior In-Context Defence (ICD) work, which suggested placing refusal demonstrations after existing conversations [1]. Using one fixed example instantiates that proposal without establishing a distinct defence mechanism. Furthermore, the original Many Shot Jailbreak paper demonstrated effective mitigation through Cautionary Warning Defense (CWD) [2] as well. The contribution therefore rests primarily on empirical validation, yet the comparison with the closest prior defences remains incomplete.

[1] Zeming Wei et al. (2023). Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations . Section 5.3

[2] Cem Anil et al. (2024). Many-shot Jailbreaking . Section 5.4 and Appendix K

Questions for authors
  1. The abstract’s claim that code is available is not substantiated: the linked repository contains only README.md and .gitignore, with no executable implementation.

  2. Table 4 describes Goal Prioritization as generating and ranking five candidate responses. This does not match the original method’s setting without training, which prioritizes safety through instructions and in-context examples. Is this a description error or a modification to the baseline? Which implementation was used for the ASR and runtime comparisons?

  3. The runtime comparison in Table 4 is insufficiently documented to substantiate the claimed efficiency advantage. Essential measurement details and controls for output length are missing, making the reported overheads difficult to reproduce or compare fairly.

Reviewer_FbMoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Safend, an inference-time defense against many-shot jailbreak (MSJ) attacks. By inserting a fixed safety-aligned refusal demonstration after the harmful examples and immediately before the target query, Safend counteracts the influence of jailbreak and reduces the attack success rate (ASR) without updating model parameters. Experiments across open-weight and API-based LLMs demonstrate its effectiveness against four context-based jailbreak attacks.

Critical strengths
  • The method requires neither model fine-tuning nor access to internal representations. Its fixed demonstration introduces minimal inference overhead, making it potentially practical for deployment with both open-weight and API-based models.
  • The method achieves substantial reductions in ASR across multiple LLMs, attack strategies, and safety benchmarks.
Critical weaknesses
  • Limited methodological novelty. Safend closely resembles existing In-Context Defense (ICD) approaches, as both rely on inserting safety demonstrations into the inference context. The main differences appear to be the number and placement of demonstrations, which seem more like configuration choices than substantial methodological innovations.
  • The theoretical analysis mainly relies on standard mathematical formulations, including attention contribution decomposition and first-order Taylor expansion. However, it does not provide sufficient causal evidence explaining why a single safety demonstration can effectively counteract numerous harmful demonstrations.
  • Safend relies on a single fixed safety demonstration adopted from prior work. The paper does not investigate whether different safety demonstrations or alternative refusal formulations would achieve similar defense performance.
  • The experiments primarily consider established jailbreak attacks without explicitly accounting for attackers aware of the defense mechanism. Since the method uses a fixed and predictable demonstration, its robustness against defense-aware adaptive attacks remains unclear.
Questions for authors
  • What fundamentally distinguishes Safend from a one-shot ICD variant with terminal demonstration placement? Beyond the number and position of safety demonstrations, could the authors clarify the methodological novelty of Safend?
  • Since ICD also relies on inserting safety-aligned refusal demonstrations, why does Safend achieve substantially better defense performance? The authors should clarify the key factors contributing to this performance gap, particularly the differences in demonstration content, number, and placement.
  • Could the authors provide stronger empirical evidence supporting the proposed theoretical analysis? For example, measuring attention contribution changes or conducting activation patching experiments could help establish a causal connection between contextual changes and safety recovery.
Reviewer_f2WjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Safend, a simple inference-time defense against many-shot jailbreaking. It inserts a fixed refusal demonstration immediately before the target query, without changing model parameters. Experiments on four LLMs show substantial reductions in attack success rates with small losses in benign task performance.

Critical strengths
  1. Safend is simple and easy to deploy. It requires no training, access to model internals, or additional model calls, and adds little inference overhead.
  2. The experiments cover both open-weight and API-based models, several contextual attacks, and benign capability benchmarks.
  3. The experiments with up to 256 harmful demonstrations also provide useful evidence that the defense remains effective as the attack context grows.
Critical weaknesses
  1. The novelty appears limited, as Safend closely resembles ICD with a refusal demonstration placed immediately before the target query. Given known recency effects, the gains may largely come from placement. A comparison with ICD at the same terminal position is needed to clarify the contribution.
  2. It is unclear whether the attacks were run against Safend directly or whether the defense was added to previously generated attack histories. Since the refusal example is fixed, the paper should test whether an attacker aware of it can work around it.
  3. The main experiments use a single fixed refusal example, leaving it unclear how sensitive the results are to this choice. The content ablation compares refusal with compliance, but does not compare different refusal examples.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.