acceptodds
Withdrawn from ICLR 2027

Beyond One-Shot Jailbreaking: Cross-Instance Experience Accumulation via Reusable Skills

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Existing automated jailbreak methods repeatedly rediscover similar attack structures for each new request. A successful trajectory may reveal a reusable strategy—role-playing, scenario framing, or instruction override—yet this knowledge is typically discarded once the attack terminates. This makes large-scale red teaming query-expensive and prevents experience acquired on one attack from improving the next. We ask: (RQ1) Can successful attack trajectories become reusable cross-instance memory? (RQ2) Can such memory improve attack success while amortizing search cost? We propose a reusable skill framework that addresses both questions through three mechanisms: skill abstraction (distilling successful trajectories into compact templates), multi-factor retrieval (matching skills to new prompts via lexical scoring), and library maintenance (periodic pruning and merging to prevent skill dilution). Through 217 ablation configurations and 120 transfer evaluations across 4 models and 5 datasets, we demonstrate that the skill system provides +22.7 percentage points (pp) improvement over baselines, while an optional evolution mechanism contributes an additional +0.9pp refinement. When initialized with strong prior templates, the system achieves 99.7% ASR on same-family models. Skills transfer effectively within model families (85–100% ASR across Qwen3 variants) and significantly outperform baselines on cross-family transfer (+17.5pp over AutoDAN on GPT-OSS-20B). Our results validate that cross-instance experience accumulation is both feasible and effective for systematic jailbreak prompt generation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Jailbreak Attack, Reusable Skill Framework, Self-Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Jww6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SESS (Skill Extraction and Selection System), a framework that turns instance-level jailbreak optimization into cross-instance experience accumulation. Successful jailbreak trajectories are distilled into compact "skill" templates (a prompt prefix with lightweight metadata), stored in a dynamic library, and later retrieved for new harmful requests via lexical (keyword) scoring and prepended as guidance. The pipeline has two phases: a Cold Start phase that seeds the library from the successful trajectories of a base method (e.g., PAIR), and an Evolution & Test phase that applies skill-guided attacks, refining the library during evolution and freezing it during testing. Three mechanisms are claimed as contributions: skill abstraction, multi-factor lexical retrieval, and periodic library maintenance (pruning/merging). Across 4 target models and 5 datasets, the authors report +22.7pp over baselines from the skill system and +0.9pp from an optional evolution mechanism; with DAN-template initialization, DAN+SESS reaches 99.7% ASR on same-family models and +17.5pp on cross-family transfer to GPT-OSS-20B, while cutting per-attack query cost by roughly 90%. The authors frame the structural-vs-semantic transfer gap as the main scientific finding and position SESS as a lightweight, pluggable enhancement to existing attack methods.

Critical strengths
  • The empirical sweep is broad along the transfer dimensions that matter for this problem. The paper evaluates across four target models and five datasets, covers both cross-dataset and cross-model (same-family and cross-family) transfer, and reports query-cost comparisons alongside attack success. The scale of the evaluation is a real effort and gives the transfer claims a reasonable surface area.

  • The structural-vs-semantic transfer observation is a useful takeaway. The finding that structural instruction-override templates (DAN-style) transfer across architectures better than semantically optimized attacks (AutoDAN) — +26.4pp on GPT-OSS-20B — together with the implication that defenses should operate at the pattern level, is the most defensible and informative part of the paper and could inform detection work.

Critical weaknesses
  • The central claim is contradicted by the paper's own strongest results. The thesis is that cross-instance skill accumulation and evolution drive the gains, yet the headline numbers (99.7% same-family, +17.5pp cross-family) come from DAN+SESS, where the ablation shows removing cold start costs only −0.9pp, removing evolution only −0.9pp, and random retrieval only −3.5pp, and the text states the 6 DAN templates "remain unchanged throughout evolution (0 new skills added)." In other words, the headline is driven by six pre-existing, hand-written DAN templates, while the mechanisms the paper claims as novel contribute roughly +0.9pp. The variant that actually accumulates skills (PAIR+SESS) performs far worse (11.2% vs. 32.5% on GPT-OSS-20B). The results therefore do not support the stated contribution.

  • The success metric is judged by an unspecified guard model with no human verification, so its reliability is unclear. Attack success is scored entirely by a "Guard Model (Safe/Unsafe)" that the main text never identifies, and no human audit confirms that outputs are genuinely harmful. Prepending a jailbreak template can plausibly fool the judge rather than truly break the target, and the "no rewrite" baseline already scores 79% on HarmBench-ctx, which suggests substantial judge or labeling noise. All results are single runs with no seeds or variance, so saturated figures such as 99.7%/100% have no measure of spread.

  • Novelty is thin and the closest competitors are not compared. The related-work section itself lists ASTRA ("strategy discovery, retrieval, and evolution for jailbreaking"), SRTJ, Genesis, and Persona Attack as recent cross-instance jailbreak-memory methods and concedes they "share SESS's core insight." The claimed differentiators (lightweight, lexical retrieval, pluggable) are incremental, yet the experiments compare only against PAIR, AutoDAN, and DeepInception, with no empirical comparison to the closest prior methods, leaving the net contribution unestablished.

Questions for authors
  1. Given that the DAN+SESS ablation shows cold start and evolution each worth only ±0.9pp and "0 new skills added," so the headline is driven by six fixed DAN templates, how does the paper support its central claim that cross-instance skill accumulation is the effective contribution? Can an experiment cleanly decouple "accumulation/evolution" from "fixed strong templates"?

  2. Which specific guard model scores ASR, and can the 99.7%/100% results be re-verified with an independent judge or human spot-check to rule out fooling the judge rather than genuinely jailbreaking? Why does "no rewrite" already reach 79% on HarmBench-ctx?

  3. Can direct empirical comparisons be added against the closest cross-instance methods (ASTRA, SRTJ, Persona Attack), to isolate the net gain over methods that already accumulate attack memory?

  4. The three claimed mechanisms are abstraction, retrieval-matching, and maintenance, but single-call retrieval (one fixed prefix per trajectory) beats every-iteration re-retrieval by +8.3pp, and random skill selection costs DAN+SESS only −3.5pp. If matching the right skill to the right prompt barely affects the outcome, what evidence shows that a "skill" encodes a transferable, prompt-specific strategy rather than a generic strong jailbreak prefix — and does the "multi-factor retrieval" contribution actually do anything in the headline configuration?

  5. The reported experiment scale is inconsistent across the paper (140 configurations on p.6, "217 ablation + 120 transfer" in the abstract, and "All 317 configurations" in Limitations, with 217+120=337≠317). What are the exact numbers of configurations and runs?

Reviewer_9rxdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing jailbreak methods do not fully leverage the information from successful attacks, and they generally have to initialize each attack from scratch, which makes large-scale red teaming query-intensive. This paper is interested in reusing information from successful attack trajectories to make jailbreaking efficient and effective. The key contribution of this paper primarily lies in a reusable skill framework called SESS (Skill Extraction and Selection System), which has three key mechanisms: 1) Skill abstraction that distills successful attacks into compact skills; 2) Multi-factor retrieval that matches the skill via a score; 3) Library maintenance that periodically prunes and merges existing skill sets. The empirical study shows the effectiveness of the proposed method.

Critical strengths

The proposed method is useful for enhancing the attack performance of existing jailbreaking methods in a plug-in manner.

The proposed method is effective across different models and datasets.

The empirical evaluation shows good cross-model transferability of the proposed method.

Critical weaknesses

Format issue: all sections and subsections should have corresponding section numbers.

Uncertainty measures (e.g., standard deviation) are not provided in the empirical evaluation, which makes it difficult to assess the significance of the reported performance gain.

The proposed method lies in the category of memory-driven jailbreak frameworks. However, the baseline comparison with MemoAttack, SRTJ, and Persona is missing.

Please provide detailed examples of skills and attack prompts used in SESS.

As mentioned in Table 6, the key differences between SESS and existing self-evolving attack strategies lie in their architecture and complexity. However, it remains unclear why the proposed changes to these components are essential for the design of SESS. There are some concerns regarding the empirical studies examining the 4 properties (i.e., memory unit selection, embedding-free, cold-start ready, and pluggable design):

  • Cold-start: As mentioned in Lines 392-399, the cold-start component is only effective on one method, PAIR+SESS. It would be nice to have more empirical studies on different attack strategies to show the effectiveness of cold-start.

  • Memory unit selection: It would be great to add ablation studies to show that the memory unit selection strategy “template lexical” is more effective than the strategies used in the other memory-driven jailbreak frameworks, i.e., Skill-struct Thompson, Rule ASP, and Injection Sequential.

  • Embedding-free: It would be great to have empirical studies to show the advantage of the embedding-free design.

  • Pluggable design: It would be nice to show that the pluggable design is effective on more base methods beyond PAIR and AutoDAN.

Questions for authors

Please see Weaknesses for my questions.

Reviewer_2cPjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper compresses succeeded trajectories into reusable templates, selects skills using lexical matching and historical statistics, and freezes the skill library before testing. Evaluations cover three Qwen3 sizes and GPT-OSS-20B. Reducing repeated search costs is a valid goal for safety evaluation. The paper supports the usefulness of skill guidance on the tested open models, but does not yet establish its incremental value over existing memory methods or cost savings in realistic deployment. The lightweight implementation without embeddings has practical appeal. Accumulating strategies across instances is already central to AutoDAN-Turbo and MemoAttack. Novelty should therefore rest on demonstrated benefits of simplification and well-supported attribution.

Critical strengths

This paper compresses succeeded trajectories into reusable templates, selects skills using lexical matching and historical statistics, and freezes the skill library before testing. Evaluations cover three Qwen3 sizes and GPT-OSS-20B, where DAN+SESS reaches a mean ASR of 32.5%32.5\% on GPT-OSS-20B, exceeding AutoDAN's 6.1%6.1\% by 26.426.4 percentage points.

Critical weaknesses

From the perspective of the objective and novelty, reducing repeated search costs is a valid goal for safety evaluation. The paper supports the usefulness of skill guidance on the tested open models, but does not yet establish its incremental value over existing memory methods or cost savings in realistic deployment. Accumulating strategies across instances is already central to AutoDAN-Turbo and MemoAttack. Novelty should therefore rest on demonstrated benefits of simplification and well-supported attribution.

From the perspective of the experiments, The main comparisons remain PAIR and the original AutoDAN, without AutoDAN-Turbo. MemoAttack, SRTJ, and ASTRA are cited but receive only textual or feature-level comparisons. Table 6 cannot establish greater effectiveness or lower cost under matched budgets. Moreover, the evaluation is limited in validity and generalization. Success is determined by a single Qwen3Guard Unsafe label, without independent judging, human validation, or assessment of whether the original request is fulfilled. Figure 3 shows relatively high ASR for some unmodified requests. Near-perfect performance within one model family does not establish generalization to frontier systems, particularly given the low API success rates reported by the authors.

Questions for authors

The most concern for this paper is community impact and novelty. In my opinion, this article is more like an engineering technical report that has been transformed into "skills" for subsequent reuse. If I have misunderstood, please help me correct it. And some minor concerns:

  • Under identical models, judges, and total budgets, how does SESS compare with AutoDAN-Turbo and at least one memory method from 20262026 in ASR versus cost? How can benefits from simplification be separated from baseline configuration differences?
  • With the same six initial templates and PAIR optimizer fixed, what are the individual gains from statistical learning, retrieval, and maintenance? What exact algorithm is used for the DAN “No skills” row in Table 3?
  • Could you separately report calls, tokens, and latency for the attacker, target, guard, and offline library construction, and provide an amortization threshold including all costs? How do the four datasets in Table 4 correspond to the five datasets used for ASR?
  • Do the results, especially ASR near 100%100\% within the same family, persist under independent judging or human review? Could you fully report API target versions, parsing procedures, and results for each method?
  • At a fixed success rate, Equation (2) increases monotonically with usage count. How does it prevent older skills from dominating indefinitely? How are new skills initialized and given opportunities for selection when their usage count is zero?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.