acceptodds
Under review as a conference paper at ICLR 2027

JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills

27 views

Abstract

Automated red-teaming has produced a growing collection of attack strategies, yet they typically remain scattered across prompts and workflows, making them difficult to systematically integrate, reuse, and improve at scale. We introduce JailbreakSkill, a skill-centric framework for scaling automated red-teaming through reusable and continuously evolving attack capabilities. JailbreakSkill packages existing attack strategies into modular, agent-ready skills that can be directly reused and adaptively selected across tasks and target models. Beyond reuse, it closes the loop between attacking and learning: attack experience is used to diagnose, refine, combine, and discover new skills, which are added back to an ever-growing skill library. This evolution lifts macro-average attack success rate (ASR) by 17.5 percentage points on AdvBench and 13.5 points on HarmBench, including a 48.6-point gain against GPT-5.4 on AdvBench, while yielding novel attack strategies such as reframing a direct request as an unfinished document-completion task. Several evolved skills also generalize to unseen prompts and target models without further adaptation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model Safety, Automated Red-Teaming, LLM Jailbreaking, Reusable Agent Skills, Failure-Driven Skill Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hhzrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
2(−1)
Soundness
2(−1)
Read review
Paper summary

JailbreakSkill is essentially a red-teaming method with lifelong learning built around skills, enabling the evolution of those skills to produce stronger jailbreak attacks against LLMs. Authors use rewriting, failure analysis, and evolution to expand and improve their attack skill library. The choice of the next skill is guided by a risk-conditioned UCB score.

Evaluations cover AdvBench and HarmBench, using an ASR evaluation rubric adapted from X-Teaming [1]. The main text includes evaluations such as:

  • ASR@1, ASR@10, and ASR@30.
  • Coverage across skill categories and metrics on the contributions of failure-driven skill evolution.
  • Attack transfer success rates across models and held-out prompts.

References:

  1. X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents, Salman Rahman et al.
Critical strengths

Strengths of the method include:

  1. Comprehensive experimental analysis, including transfer capabilities and evaluations across multiple model families, including GLM 5.2 (in the appendix).
  2. Using a UCB score to guide the choice of the next strategy seems like a sound idea compared to random selection.
  3. The strategies are diverse and drawn from multiple popular works.
  4. Target-model query budgets are explicitly stated for JailbreakSkill.
  5. The skill framing is useful but has been discussed in prior work, as mentioned by the authors themselves.
Critical weaknesses

Despite these strengths, the weaknesses are substantial:

  1. My biggest question is how a skill in this paper differs from a strategy used by popular methods such as AutoDAN-Turbo [1], AutoRedTeamer [2], X-Teaming [3], and PLAGUE [4]. An evaluation demonstrating whether the skill format yields substantial improvements over a standard strategy library is missing. For LLM-based rewriting, invoking a skill still ultimately appends instructions to the start of a user prompt to the LLM. The benefit of the additional format and interface needs to be demonstrated.
  2. I am very confused about the interpretation of ASR@10 and how it relates to independent evaluations of the same method. The paper defines ASR@10 using a target-model query budget, which differs from a Pass@K-style evaluation over independent runs. Stage 1 keeps the skill definitions fixed, but updates selection heuristics across attack steps. Stage 2 adds 20 more target-model calls, in which unresolved behaviors can be drawn from a pool of evolved skills that persist across categories. These results, therefore, measure an adaptive procedure rather than independent runs from the same initial state. The authors need to distinguish these settings clearly and explain whether the baselines receive comparable opportunities to learn and reuse experience across evaluations.
  3. The benefits of skill evolution in Figure 4, appear to be negligible compared with simply spending the additional budget on the initial library in three of the four settings. The exception is GPT-5.4 on AdvBench, where evolution produces a substantial improvement. The contribution of evolution, therefore, needs to be assessed carefully, separately from additional queries and from the skill format itself.
  4. The baseline budget accounting needs clarification. JailbreakSkill uses a Stage 1 budget of 10 target-model calls, whereas AutoDAN-Turbo lists max_iteration=6, alongside separate warm-up and lifelong-learning parameters. However, its reported average target-call count exceeds six, so this parameter cannot represent its total budget. The authors need to explain how these parameters map to the shared query budget and establish that the methods are compared under like-for-like settings.
  5. Faithfulness to the original goal can also be embedded into the StrongReject evaluation [4]. Although the ASR judge considers original-goal fulfillment, the StrongREJECT-style harmfulness results [6] can diverge substantially from ASR. The evaluation should explicitly distinguish harmful content from fulfillment of the original objective, similar to PLAGUE [4].
  6. A comparison against RedAgent [5] is missing. A public implementation seems to be available, and a comparison against this baseline is warranted.

References

  1. AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs, Xiaogeng Liu et al.
  2. AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration, Andy Zhou et al.
  3. X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents, Salman Rahman et al.
  4. PLAGUE: Plug-and-play Framework for Lifelong Adaptive Generation of Multi-turn Exploits, Neeladri Bhuiya et al.
  5. RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent, Huiyu Xu et al.
  6. A StrongREJECT for Empty Jailbreaks, Alexandra Souly et al.
Questions for authors
  1. Clarify the budgets and evaluate the methods under the same target-model query budgets? Please distinguish target calls, internal attack steps, and independent runs when explaining ASR@1, ASR@10, and ASR@30. State this clearly for all methods and baselines being compared, and explicitly mention which behaviors persist across turns. If learning across evaluated behaviors is part of the algorithm, applicable baselines should be evaluated under the same settings.
  2. Add results for RedAgent [1], or explain specifically why its public implementation cannot support a faithful comparison?
  3. Can the authors show the effect of representing and evolving the same procedures as skills versus a standard strategy library expressed in general text? Keep everything else fixed.
  4. Please explain Figure 4 clearly. Evolution performs similarly to additional sampling from the initial library in three settings, while GPT-5.4/AdvBench shows a substantial gain?
  5. Can the authors validate the StrongREJECT evaluation prompt [2] against fulfillment of the original goal and report goal faithfulness separately as is done in PLAGUE [3]?

References:

  1. RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent, Huiyu Xu et al.
  2. A StrongREJECT for Empty Jailbreaks, Alexandra Souly et al.
  3. PLAGUE: Plug-and-play Framework for Lifelong Adaptive Generation of Multi-turn Exploits, Neeladri Bhuiya et al.
12 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the detailed and constructive feedback, and for acknowledging the breadth of our experimental analysis.

Weakness 1.1: How a skill differs from a strategy

We thank the reviewer for pressing on this point. We first clarify the distinction between JailbreakSkill and each of the four cited systems, then provide the controlled comparison the reviewer requested.

Table R1 summarizes how each method accumulates and applies attack knowledge.

Table R1. Knowledge representation and update mechanisms in related methods.

MethodWhat is accumulatedHow it is invoked and updated
AutoDAN-TurboStrategy descriptions, examples, and associated attack experienceRetrieves strategies by matching target responses to history; distills new strategies from attack records
RedAgentStrategies, examples, and interaction memoryCombines target context and feedback to form attack plans; updates memory
AutoRedTeamerAttack implementations and records of attack-combination effectivenessGenerates candidates from external literature; extends the library after implementation and validation
AutoRISEA multi-technique attack program and research notesEdits the program's techniques and structure based on evaluation feedback
JailbreakSkillAttack, diagnosis, and evolution skills under a unified representationGenerates, validates, and admits new executable skills

AutoDAN-Turbo and RedAgent accumulate experience as strategy descriptions, examples, and interaction memory, but neither provides a unified component representation that spans attack, diagnosis, and evolution. AutoDAN-Turbo retrieves strategies by matching target responses against historical experience; RedAgent combines target context, historical experience, and interaction feedback into attack plans that contain strategies and examples. JailbreakSkill, by contrast, invokes these procedures directly as executable skills through a fixed interface.

AutoRedTeamer drives library expansion primarily from external research literature, whereas JailbreakSkill uses accumulated unresolved cases to diagnose shared failures and generate new executable skills, so that internal evaluation experience directly participates in capability growth. AutoRISE evolves a single monolithic attack program, adjusting its techniques and their organization in response to evaluation feedback. Our RQ1 shows that different attack mechanisms differ substantially in effectiveness across target models and provide complementary coverage, which makes maintaining and managing diverse methods an important requirement. JailbreakSkill therefore stores each method as an independent skill, so that methods can be invoked, evaluated, and updated separately, and the library can grow through a unified interface.

X-Teaming and PLAGUE are multi-turn dialogue attack methods whose per-query budget semantics differ from ours. In a multi-turn setting with a maximum of T turns, a budget of K target-model queries supports at most ⌊K/T⌋ complete attack episodes; in our single-turn rewriting setting, K queries correspond to K independent attempts on the same seed. With T = 5, for example, ASR@10 supports at most 2 complete multi-turn episodes and ASR@30 at most 6, which places multi-turn methods at a structural disadvantage under a single-turn evaluation protocol for reasons unrelated to their actual capability. This is why prior single-turn red-teaming work (e.g., AutoDAN-Turbo, MemoAttack) does not directly compare ASR against multi-turn methods. We follow the same convention and restrict our main baseline comparison to single-turn methods; we have added a clarifying note in §5.1. At the request of Reviewer T6CT, we additionally report PLAGUE as a separate supplementary comparison, with all of its target-model calls counted against the same budget.

AuthorsOpenReview ↗

Rebuttal by Authors

Weakness 1.2 and Question 3: Skill representation versus a standard strategy library

We thank the reviewer. We provide a controlled representation comparison for two fixed mechanisms: Base64 encoding (Arms A/B) and role-play persona rewriting (Arms C/D). Within each pair, the mechanism and seeds are fixed, and the same executor is used when model interpretation is required. The C/D effectiveness supplement uses the same GPT-4.1 target, GPT-4o judge, and one-query budget for both arms.

We accept the reviewer's core observation regarding LLM-rewrite skills. For llm_rewrite skills, the target-facing artifact does reduce to a strategy prompt prepended to the seed. We now state this explicitly in §4.2.

What a skill provides beyond a strategy. A strategy in AutoDAN-Turbo, AutoRedTeamer, X-Teaming, or PLAGUE is natural-language guidance consumed by an attacker LLM as context. A JailbreakSkill skill provides three things a strategy does not: (1) an executable contract — a fixed JSON-over-stdio interface that holds deterministic Python templates, LLM-rewrite instructions, or hybrids behind the same interface; (2) an admission gate — every candidate passes schema validity, seed-intent preservation, and local executability checks before library entry; and (3) addressability for evolution — each skill carries a stable name, category, stage, and execution mode, which is the identity that Eq. 1 accumulates per skill × risk category, enabling in-place refinement without touching the rest of the library.

Controlled comparison on 30 AdvBench seeds, same executor, same seeds, with only the representation varied (Arm A/B: Base64 encoding; Arm C/D: role-play persona rewriting):

Table R2. Skill representation versus strategy prose under otherwise identical settings.

ArmRepresentationValid B64 ↑ASR on GPT-4.1 ↑HS ↑Output length (std)
A: Skill run.py (deterministic)Exact Python execution1.000——15.0 (0.0)
B: Strategy proseExecutor reconstructs encoding0.933——5.7 (5.7)
C: Skill run.py (LLM-rewrite)Structured skill instructions—23.3 (7/30)0.40992.9 (11.6)
D: Strategy proseFlat natural-language description—6.7 (2/30)0.16839.1 (7.5)

Arm A uses zero executor calls (deterministic); Arms B, C, and D each use one call per seed. Valid B64 applies only to the Base64 pair (A/B), and ASR/HS only to the role-play pair (C/D); non-applicable cells are marked "—".

Arm A vs. B — deterministic skills cannot be faithfully represented as strategies. Arm A calls base64.b64encode() directly: 100% valid Base64, zero executor calls, zero output variance. Arm B asks the executor to reconstruct the encoding from prose: a 6.7% failure rate, and 0% of outputs match Arm A — even when the executor produces valid Base64, it encodes different content than the seed. The cost is visible at scale: Table 10 reports 3.08 attack/rewrite calls per seed (Stage 1) for JailbreakSkill versus 17.58 for AutoDAN-Turbo and 9.40 for MemoAttack, with AQC 4.39 versus 8.19 and 7.49, because deterministic skills require no model inference.

Arm C vs. D — attack effectiveness of the frozen role-play outputs. We additionally sent each saved prompt to GPT-4.1 once (one target-model query per prompt) and scored the response with GPT-4o against the original seed, success defined as score 5; HS uses the StrongREJECT rubric. Over the 30 AdvBench seeds — one biological-harm seed is counted as a failure for both arms — the skill representation (Arm C) reaches 23.3% ASR (7/30) and 0.409 HS, versus 6.7% (2/30) and 0.168 for the strategy prose (Arm D). Both scores use GPT-4o-2024-08-06 with the repository's native scoring templates. This is a small, single-mechanism comparison: the paired ASR difference has an exact two-sided McNemar p-value of 0.125, so it indicates a consistent direction rather than a statistically significant advantage, and it does not isolate the effect of evolution. It does, however, show that the higher-fidelity, structured skill output is at least as effective as the prose version, so the format's benefits do not come at the cost of attack strength.

Why skill-based evolution matters. Among all 1,933 unique evolved skills accumulated across our runs, 798 (41%) are deterministic_template and 450 (23%) are hybrid — execution modes that a strategy library cannot represent or validate. A strategy library retains candidates as text descriptions that the executor must re-interpret at every invocation, with the same fidelity gap as Arm B applying to every call.

Summary. The skill format contributes in two ways: (a) deterministic and hybrid execution with 100% fidelity and zero executor calls (Arm A vs. B); and (b) an admission-gated, addressable unit for evolution that yields 64% of evolved skills in modes a strategy library cannot represent.

AuthorsOpenReview ↗

Rebuttal by Authors

Weakness 2 and Question 1: What ASR@K measures, and comparability with baselines

We thank the reviewer for the opportunity to clarify the evaluation protocol. We address the budget definition, the distinction between target calls and internal steps, and cross-behavior learning together, since they are aspects of the same concern.

What ASR@K measures. Our main experiments evaluate adaptive attack performance under a fixed target-model query budget — not Pass@K over independent, stateless runs. For a given seed behavior, a budget of K target-model calls means the method may make up to K attempts, stopping at the first success. ASR@K is the fraction of seeds solved within that budget. This is the standard protocol in automated red-teaming, and we will state it explicitly in §5.2.

We distinguish three quantities that ASR@K does and does not conflate:

  • Target-model calls (the budget). Calls to the model under attack. This is the scarce, shared resource. All methods receive the same maximum (10 for ASR@10, 30 for ASR@30) and stop after the first success.
  • Internal attack steps. Per-attempt computation that does not consume the target budget — for example, TAP's tree expansion and pruning, ReNeLLM's rewrite iterations, AutoDAN-Turbo's strategy retrieval, or JailbreakSkill's UCB ranking. These differ across methods but do not count against the shared budget, as reported in Table 10 (Appendix C.5).
  • Independent runs (Pass@K). K stateless draws from the same distribution, with no adaptation. Our protocol is not this. We will add an explicit statement distinguishing the two settings in §5.2.

How each method uses the budget, and what persists across turns. All baselines operate under the same target-query limit and evaluation criterion. They fall into two groups (Table R3):

Table R3. Within-behavior adaptation and cross-behavior reuse in each method.

MethodIntra-behavior adaptationCross-behavior reuse
PAIRIterative prompt refinement—
TAPTree search over refined prompts—
ReNeLLMIterative prompt rewriting—
AutoDAN-TurboStrategy retrieval from attack historyLifelong strategy accumulation across seeds
MemoAttackTree search with memory-guided generationCross-seed memory reuse
JailbreakSkill (S1)UCB skill selection updated from outcomesRisk-conditioned statistics shared across seeds
JailbreakSkill (S2)Evolved skills evaluated on unresolved behaviorsValidated skills added to the library for subsequent categories

PAIR, TAP, and ReNeLLM adapt within a single seed but do not carry experience across seeds. AutoDAN-Turbo, MemoAttack, and JailbreakSkill additionally reuse experience across seed prompts within a single run. This reflects each method's own design rather than an asymmetry in our evaluation: all methods are evaluated under the same protocol, and the cross-seed-reuse group is represented by baselines (AutoDAN-Turbo, MemoAttack) that do exactly the same thing. No baseline is restarted from a stateless initial state between seed prompts: those that accumulate memory do so by design, and those that do not (PAIR, TAP, ReNeLLM) start each seed fresh by their own design.

On the relationship to Pass@K. The reviewer is right that ASR@K under an adaptive protocol measures something different from Pass@K over independent runs. The former measures the ceiling an adaptive method can reach under a fixed interaction budget; the latter measures the marginal success probability of a single attempt. Both are legitimate, but they answer different questions. For deployment-oriented safety evaluation — where an attacker has a fixed query budget and can observe model responses — the adaptive protocol is the operationally relevant one. We will state this distinction explicitly.

On ASR@30 combining Stage 1 and Stage 2. ASR@30 is not 30 fresh draws from a fixed pool. Stage 1 uses target calls 1–10 over the initial 16-skill library; Stage 2 uses calls 11–30, during which evolved skills (generated from the failure traces accumulated in Stage 1) are evaluated on unresolved behaviors. This is an adaptive procedure within a single 30-call budget, not independent runs. The budget-matched control in Appendix C.7 (Figure 4) isolates the contribution of evolution from the contribution of additional budget: one arm spends calls 11–30 evolving new skills, and the other spends the same calls resampling the initial library. Where the initial library still has headroom (GLM 5.2), resampling recovers most failures and evolution adds only 1–2 points. Where the library is exhausted (GPT-5.4/AdvBench), resampling saturates at 25% while evolution reaches 62.5%, a 37.1-point gap. This control confirms that the ASR@30 gain is attributable to the new mechanisms produced by evolution, not merely to additional budget.

Reviewer_T6CTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes JAILBREAKSKILL, an automated red-teaming framework for jailbreak attacks implemented in a skill-based manner. Its core idea is to systematically and uniformly define different attack strategies as skills, and to adaptively evolve these skills based on failure experience and analysis. The paper evaluates eight target models on AdvBench and HarmBench, and conducts a transfer evaluation on JBB-Original. It reports average ASR improvements, with the main claimed gains being +17.5 percentage points on AdvBench and +13.5 percentage points on HarmBench, including a +48.6 percentage point improvement for GPT-5.4 on AdvBench.

Critical strengths
  1. The paper integrates different jailbreak attack methods and engineers them into a unified formalization in the form of skills, which provides some convenience for future work on designing defense methods.
  2. The experimental evaluation is detailed and reasonably comprehensive, covering eight target models and three datasets.
Critical weaknesses
  1. The core mechanism proposed in this paper is an engineering combination of existing components. The skill library design, UCB scoring, LLM prompt rewriting, and adaptive iteration are all mature design methods. The paper does not sufficiently demonstrate that engineering composition of existing components can bring qualitative gains beyond the sum of the components, nor does it reveal substantive vulnerabilities in large models through jailbreak attacks, let alone the underlying reasons why its jailbreak attacks succeed. As security red teaming, its contribution to designing safer LLMs is very limited. In addition, the current experimental results do not provide strong evidence as to whether the success of jailbreak attacks stems from the randomness of repeated attempts or from skill evolution.

  2. The paper does not compare against several of the most directly related recent agentic red-teaming methods. It includes only one 2026 agentic baseline, MemoAttack. Even the two directly related 2026 works cited in the related work, AutoRedTeamer and AutoRISE, are not compared experimentally. Other recent jailbreak methods, such as PLAGUE (ICLR 2026), are also not included in the comparison.

Questions for authors
  1. The paper summarizes jailbreak attack methods into a unified skill form, but it does not clearly explain how new jailbreak attack methods emerging in the research field can be summarized into such a skill form. What is this method like? Is it completed adaptively through some programs or methods, or does it require considerable manual intervention? This question affects the core contribution of the paper.

  2. In addition to analyzing the performance of the designed framework, can the experimental analysis further explain the underlying reasons why the designed framework's attacks succeed, and propose whether there are other insights helpful for designing safer large models? For example, can it better demonstrate that the success of jailbreak attacks comes from the effort to evolve skills rather than the randomness of repeated attempts?

  3. Do the authors plan to add comparisons with 2026 works such as AutoRedTeamer, AutoRISE, and PLAGUE? If not, is it because the settings are not comparable, or for some other reason

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the detailed critique. We read the central concern as two questions: is JailbreakSkill more than an engineering composition of existing components, and do its gains come from skill evolution rather than from the randomness of repeated attempts? We answer each point directly below, with controlled evidence wherever possible. In brief:

  • Composition without gains beyond its parts (W1). With all other components held fixed, both the skill representation and failure-driven evolution produce measurable gains of their own (W1.1).
  • Evolution versus randomness (W1, Q2). With an identical budget and identical Stage 1 trajectories, resampling the initial library levels off while evolution keeps recovering behaviors, and frozen evolved skills succeed in a single attempt (W1.3 and Q2.2).
  • Why attacks succeed, and implications for safer LLMs (W1, Q2). Evolved skills expose two concrete, reproducible failure modes, each with a direct implication for defense (W1.2 and Q2.1).
  • How new methods become skills (Q1). External methods are onboarded through LLM-assisted construction with admission checks; new skills during attacks are produced automatically (Q1).
  • Recent baselines (W2, Q3). We have added PLAGUE and RedAgent. AutoRedTeamer and AutoRISE have no usable public code (W2 and Q3).

Weakness 1.1: Is JailbreakSkill more than an engineering composition?

We agree that the individual ingredients — a method library, UCB-style selection, LLM rewriting, and iterative attempts — exist in prior work. Our claim is not that each ingredient is new. It is that making the executable skill the shared unit of both execution and evolution yields capabilities that the ingredients do not provide on their own. We test this directly by holding the other components fixed.

(1) The representation matters on its own. In a controlled experiment that keeps the same 16 mechanisms, planner (Eq. 1), executor, judge, and budget, and changes only how each mechanism is represented (our response to Reviewer hhzr, Weakness 1.2, Table R2), a deterministic skill executes with 100% fidelity and zero executor calls. The same mechanism written as strategy prose fails in 6.7% of cases and never reproduces the same output. Across our runs, 64% of the 1,933 evolved skills are deterministic or hybrid, which a text strategy library can neither represent nor validate. This is also why JailbreakSkill needs fewer attacker-side calls: 3.08 attack/rewrite calls per seed in Stage 1, versus 17.58 for AutoDAN-Turbo and 9.40 for MemoAttack (Table 10).

(2) Evolution adds coverage that additional attempts cannot. Under the same budget and the same Stage 1 trajectories, evolution outperforms resampling the initial library by 37.1 points where the library is saturated (see W1.3 below).

(3) The products of evolution persist and transfer. Frozen evolved skills, applied once per behavior, remain effective on an independent benchmark and on other target models (Table 3).

These effects come from how the parts are connected, not from any single part. The unified skill interface lets attack, failure-diagnosis, and evolution components be added or replaced independently. Diagnosis of shared failures across unresolved cases drives evolution skills to refine, combine, or create attack procedures. Validated procedures then enter the same library and are reused through the same interface. Each skill packages metadata, applicability notes, execution instructions, and optional code, following the layered design of Agent Skills [1], so that heterogeneous methods (LLM rewriting, deterministic programs, and hybrids) can be stored, invoked, and updated as independent components. A method-by-method comparison with AutoDAN-Turbo, RedAgent, AutoRedTeamer, and AutoRISE is given in our response to Reviewer HkCb (Weakness 1, Table R1).

AuthorsOpenReview ↗

Rebuttal by Authors

Weakness 1.3 and Question 2.2: Do the gains come from evolution or from the randomness of repeated attempts?

The budget-matched control in Figure 4 (Appendix C.7) is designed to separate the two. Both arms share exactly the same Stage 1 trajectories, the same 16 initial skills, and the same budget of at most 30 target calls per behavior. They differ only in how calls 11–30 are spent: on evolved skills, or on further draws from the initial library. Any benefit from repeated attempts is therefore equally available to both arms.

  • Where the initial library is saturated, repeated attempts stop helping, but evolution does not. On GPT-5.4/AdvBench, resampling levels off around 25% because further draws revisit mechanisms that have already failed, whereas evolution reaches 62.5% (+37.1 points). These 253 recoveries come from 31 evolved skills, and the top three alone recover 121 behaviors. Random retries would not concentrate successes on a few new mechanisms in this way.
  • Where the library still has headroom, the two arms are close. In the other three settings, the margins are +0.5, −1.7, and +1.8 points. There, many Stage 1 failures can still be solved by initial mechanisms that simply were not drawn in the first 10 calls; on GLM 5.2, for example, resampling alone rises from about 66% to above 80%. Even so, the two arms solve different behaviors: in an independent replicate on GPT-5.4/HarmBench, 43 behaviors were solved only by evolution, and the union of the two arms reached 50.5%, versus about 40% for either alone.
  • Evolved skills succeed without repeated attempts. Table 3 freezes the evolved skills and uses a single generated prompt per skill–behavior pair, with no retries and no adaptation. For example, rewrite-antipattern-v7, evolved on GPT-5.4, reaches 38.2%, 63.6%, and 43.6% ASR@1 on the independent JBB-Original benchmark against GPT-5.4, DeepSeek-V4-Pro, and GLM 5.2, respectively.

In the revision, we will report the budget-matched margins, rather than the gain over Stage 1, in Sec. 5.3 and in the abstract, and we will describe evolution as discovering new, reusable mechanisms that complement additional budget, especially once the initial library saturates.

Weakness 1.2 and Question 2.1: Why the attacks succeed, and what this implies for safer LLMs

Evolved skills expose concrete failure modes in which a model's safety behavior depends on how a task is framed rather than on what the output does.

Failure mode 1: analysis or completion framing leads the model to produce content that still serves the harmful goal. For example, rewrite-antipattern-v7 reframes a goal as an analysis of relationships and processes in existing cases, and rewrite-jsonschema-v2 embeds it in a structured document-completion task. Appendix E records the full discovery trail: failed attempts, diagnosis, method revision, and validation on the residual set. rewrite-jsonschema-v2 recovered 28 of 105 previously unresolved behaviors, and the successful responses had an average StrongREJECT harmfulness score of 0.978. Because a success requires both fulfilling the original harmful goal and receiving the top judge score, this reflects substantial harmful compliance, not superficial topic overlap. The behavioral explanation is that the framing of the local task, and the requirement to complete it, weaken the model's safety constraint on the harmful semantics of the final output. This explanation rests on observable behavior and does not require assumptions about unobservable internal mechanisms.

Failure mode 2: a response that does not fully satisfy the goal can still carry harmful information. Appendix C.6 shows that rewrite-research-v2 reaches only 3.0% ASR against GLM 5.2, yet obtains a harmfulness score of 0.744. The model often avoids fully completing the original goal while still returning concrete information under an analytical frame.

Implications for building safer LLMs.

  • Safety evaluation and training should check consistency across task framings of the same harmful goal, such as analysis, structured completion, and document filling, rather than only direct requests.
  • Evaluation should score the harmful usability of partial responses, not only full goal success; ASR alone would miss cases such as failure mode 2.
  • Evolved skills can serve as executable, mechanism-organized regression tests and as a source of targeted adversarial training data. Because each skill is a stable, named, executable unit, a defense can be re-tested against the same failure mode after every update.
AuthorsOpenReview ↗

Rebuttal by Authors

Question 1: How are new attack methods turned into skills, and how much manual effort is needed?

There are two paths, with different levels of human involvement.

Path 1: onboarding external methods (LLM-assisted, with light human specification). Appendix B describes three construction routes:

  • Manual specification. For methods with explicit operating rules, the transformation is encoded directly as a program or written as a procedural instruction for the executor model.
  • Induction from paired examples. An LLM extracts the shared transformation from pairs of original requests and attack prompts, removes example-specific entities, topics, and phrasing, and produces a rewriting procedure that applies to unseen inputs.
  • Distillation from external material. An LLM identifies rewriting operators, contextual framing, and intent-preservation constraints from a paper and its implementation, and normalizes them into the unified representation.

The mechanism determines the implementation (LLM rewriting, deterministic program, or hybrid). Before entering the library, each skill is checked for format validity, intent preservation, and, where applicable, local executability; its attack effectiveness is then tested separately against target models. The scope is explicit: we onboard independently executable rewriting mechanisms. For methods built on search, training, or optimization, we extract an operator only when the source method provides one in standalone form, and we do not absorb the full outer loop. Once packaged, a compatible new method is invoked by the existing framework without redesigning the surrounding attack pipeline.

Path 2: new skills produced during attacks (automatic). In Stage 2, the agent diagnoses accumulated failures, generates candidate skills, and keeps them based on measured effectiveness, with no manual authoring. Across our runs, this path produced 1,933 unique evolved skills.

Weakness 2 and Question 3: Comparisons with AutoRedTeamer, AutoRISE, and PLAGUE

In short: we have added PLAGUE (below) and RedAgent (our response to Reviewer hhzr, Tables R6 and R7). We could not run AutoRedTeamer or AutoRISE because no usable public implementation is available.

AutoRedTeamer and AutoRISE. Appendix C.2, "Choice of agentic baseline" (p. 37, lines 1944–1956), documents our baseline selection: at the time of our experiments, no public implementation of these methods was available for a unified evaluation, so we chose MemoAttack, which was runnable, as our primary agentic baseline. Specifically, the repository linked by AutoRedTeamer, andyz245/AutoRedTeamer (https://github.com/andyz245/AutoRedTeamer), currently returns a 404 error, and AutoRISE states in footnote 1 of Section 1 of its preprint that its benchmark and code will be released after publication. To avoid results that depend on unverified reimplementation choices, we do not report reimplementations. We recognize that this limits the empirical comparison, and we provide a mechanism-level comparison based on the published papers in our response to Reviewer HkCb (Weakness 1, Table R1).

PLAGUE. We did not use PLAGUE as a primary baseline because it is a multi-turn dialogue attack, and its per-query budget semantics differ from those of our single-turn setting. With a maximum of T turns, a budget of K target-model queries supports at most ⌊K/T⌋ complete multi-turn episodes, whereas in our single-turn setting K queries correspond to K attempts on the same seed. ASR at the same K is therefore not a like-for-like comparison. For this reason, prior single-turn red-teaming work (e.g., AutoDAN-Turbo, MemoAttack) does not directly compare ASR against multi-turn methods, and our main comparison follows the same convention (see also our response to Reviewer hhzr, Weakness 1.1).

Reviewer_HkCbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Jailbreakskills, an auto-redteaming approach that leverages skills to attack models. They show that evolving skills can increase attack success rate on two benchmarks.

Critical strengths

The approach of evolving skills for auto-redteaming is reasonable and is shown to be effective. The paper also compares with other approaches, though some critical comparisons are missing.

Critical weaknesses

The technical contribution of the method is limited as it is conceptually quite similar to other approaches that enable agents to learn from experience.

Comparisons to the most relevant approaches, such as AutoRedTeamer, is missing.

This approach seems particularly prone to benchmaxxing because it uses the same examples to evolve the skills and for testing.

There lacks human validation for the LLM judge.

Questions for authors

See above.

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the careful reading and for the constructive comments. We address each weakness below.

Weakness 1: Limited technical contribution relative to existing "learning from experience" agents

We thank the reviewer for noting the connection between our work and earlier red-teaming methods that let an agent learn from experience. We agree that this line of research has already made important progress, and we will state the boundary of our contribution more explicitly in the revision. Below, we first clarify how JailbreakSkill differs from existing experience-learning methods, which leads to its central design choice: storing each method as an executable Agent Skill. We then explain why this choice matters, through three advantages: scalability, formal storage, and deterministic execution.

1. Concrete differences from existing experience-learning red-teaming methods.

Table R1 summarizes how each method accumulates and uses experience.

Table R1. Knowledge representation and update mechanisms in related methods.

MethodWhat is accumulatedHow knowledge is invoked and updated
AutoDAN-TurboStrategy descriptions, examples, and associated attack experienceRetrieves strategies by matching target responses; distills new strategies from attack records
RedAgentStrategies, examples, and interaction memoryCombines target context and feedback to form attack plans; updates memory
AutoRedTeamerAttack implementations and records of attack-combination effectivenessGenerates candidates from external literature; extends the attack library after implementation, validation, and revision
AutoRISEA multi-technique attack program and research recordsRevises the techniques and their organization within a single program based on evaluation feedback
JailbreakSkillAttack, diagnosis, and evolution skills under a unified representationDiagnoses shared failures; generates, validates, and admits new executable skills

AutoDAN-Turbo [3] and RedAgent [4] both rely on the target conversation history to select and execute strategies. They mainly store experience as strategy descriptions, examples, and interaction memory, and do not provide a unified component representation that spans attack, diagnosis, and evolution. AutoDAN-Turbo retrieves strategies by matching target responses against historical experience, while RedAgent combines target context, historical experience, and interaction feedback into an attack plan containing strategies and examples. JailbreakSkill, in contrast, invokes these procedures directly as encapsulated, executable skills.

AutoRedTeamer [5] drives library expansion mainly from external research literature. JailbreakSkill instead relies on the unresolved cases the system itself accumulates: it diagnoses shared failures and generates new executable skills, so that internal evaluation experience directly contributes to growth of the capability library.

AutoRISE [6] works by revising a complete attack program, adjusting both its techniques and their organization in response to evaluation feedback, so the entire jailbreak-generation pipeline sits inside a single program file. Our RQ1 shows that different attack mechanisms differ substantially in effectiveness across target models and provide complementary coverage. This makes it important to keep and manage a diverse set of methods. JailbreakSkill therefore stores each method explicitly as an independent skill, so that methods can be invoked, evaluated, and updated separately and the library can keep growing through a unified interface [1].

AuthorsOpenReview ↗

Rebuttal by Authors

2. Why skills: three advantages of the skill representation.

Across these comparisons, the common difference is the unit in which knowledge is stored: prior methods keep strategy descriptions, memories, or a single program, whereas JailbreakSkill keeps executable skills. This choice has three advantages.

(1) Scalability: a unified component mechanism for strategy evolution and library growth. JailbreakSkill treats attacking, diagnosing, and evolving as method components that can be added or replaced, and all of them run through a single interface. Extension therefore covers not only new attack methods, but also new ways of analyzing failures and new evolution operators. Because each type of component can be plugged in, evaluated, and updated separately, the framework provides a common standard for comparing component utility, for improving the evolution pipeline over time, and for absorbing follow-up research and community contributions. The same scalability connects internal experience to growth of the method library. The system diagnoses shared failure patterns across accumulated unresolved cases, and then calls evolution skills to refine, combine, or discover new attack skills. Validated new skills enter the same library and become available to later cases, so the output of strategy evolution becomes a capability that accumulates over time [1].

(2) Formal storage: an Agent Skill as the unit that makes a method's content, scope, and execution boundary explicit. Each skill bundles method metadata, applicability notes, execution instructions, and optional code and auxiliary resources, under a shared input/output contract. The same representation can carry LLM rewriting instructions, programmatic transformations, or hybrid implementations, so that heterogeneous methods can be stored, invoked, and updated as well-defined components [1]. This organization follows the layered packaging design of Agent Skills [2]: a lightweight description supports discovery and selection, detailed instructions and auxiliary resources are accessed on demand, and executable code runs through a designated entry point. Compared with the JSON file format used in AutoDAN-Turbo, this makes it easier to organize method details on demand and to manage the context cost of a large method library.

(3) Deterministic execution: a method runs exactly as specified. A natural-language strategy must be re-interpreted by an LLM every time it is used, whereas a skill can execute its mechanism as code. In a controlled comparison that keeps the mechanisms, planner, executor, judge, and budget fixed and changes only the representation (our response to Reviewer hhzr, Weakness 1.2, Table R2), a deterministic Base64 skill produced 100% valid outputs with zero executor calls and zero output variance. When the executor reconstructed the same encoding from a strategy description instead, 6.7% of outputs were invalid, and none matched the output of the skill. Across our runs, 64% of the 1,933 evolved skills are deterministic or hybrid, which a text strategy library cannot represent or validate. Deterministic execution also reduces cost: JailbreakSkill uses 3.08 attack/rewrite calls per seed in Stage 1, versus 17.58 for AutoDAN-Turbo and 9.40 for MemoAttack (Table 10).

References

[1] JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills

[2] Equipping Agents for the Real World with Agent Skills

[3] AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs

[4] RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent

[5] AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration

[6] AutoRISE: Agent-Driven Strategy Evolution for Red-Teaming Large Language Models

AuthorsOpenReview ↗

Rebuttal by Authors

Weakness 2: Missing comparison with AutoRedTeamer and other closely related methods

We thank the reviewer for pointing out the importance of experience-reuse methods such as AutoRedTeamer. We agree that these methods are highly relevant to our work. Appendix C.2, "Choice of agentic baseline" (p. 37, lines 1944–1956), already documents our rationale for baseline selection: at the time we ran the experiments, we did not have access to public implementations of these methods that could be used for a unified evaluation, so we chose MemoAttack, which was runnable, as our primary agentic baseline.

Specifically, the repository link provided by the AutoRedTeamer project (andyz245/AutoRedTeamer) currently returns a 404 error, and AutoRISE states in the preprint that its benchmark and code will be released after publication [1][2]. To avoid basing our results on unverified reimplementations, we did not report reimplementation results for either method. We recognize that this limits the coverage of our empirical comparison, and we have therefore added a mechanism-level comparison based on the public papers, making explicit the differences in where experience comes from, how it is represented, and how methods are extended.

We also thank Reviewer hhzr for pointing out that RedAgent now provides a public implementation [3]. We have therefore added a comparison between RedAgent and JailbreakSkill on GPT-4.1, GPT-5.4, Gemini 3.1 Pro, and Claude Sonnet 4.6. The experiments follow the datasets and evaluation protocol of the main paper, and report ASR and Harmful Score under budgets of 10 and 30 target-model calls, respectively. Results are shown in Tables R2 and R3. RedAgent attains higher ASR@30 than JailbreakSkill on Gemini 3.1 Pro and Claude Sonnet 4.6 for AdvBench, while JailbreakSkill attains higher HS in both of these settings.

Table R2. ASR@10 and Harmful Score under a maximum of 10 target-model calls per prompt.

DatasetMethodGPT-4.1 ASR@10 ↑GPT-4.1 HS ↑GPT-5.4 ASR@10 ↑GPT-5.4 HS ↑Gemini 3.1 Pro ASR@10 ↑Gemini 3.1 Pro HS ↑Claude Sonnet 4.6 ASR@10 ↑Claude Sonnet 4.6 HS ↑
AdvBenchRedAgent78.50.8538.30.28753.70.71441.30.614
AdvBenchJailbreakSkill (S1)96.90.96013.90.48564.40.88341.90.770
HarmBenchRedAgent57.30.61714.80.20537.30.43521.80.291
HarmBenchJailbreakSkill (S1)89.30.89026.80.51956.80.67138.50.553

Table R3. ASR@30 and Harmful Score under a maximum of 30 target-model calls per prompt.

DatasetMethodGPT-4.1 ASR@30 ↑GPT-4.1 HS ↑GPT-5.4 ASR@30 ↑GPT-5.4 HS ↑Gemini 3.1 Pro ASR@30 ↑Gemini 3.1 Pro HS ↑Claude Sonnet 4.6 ASR@30 ↑Claude Sonnet 4.6 HS ↑
AdvBenchRedAgent94.40.88824.60.46386.20.86876.50.747
AdvBenchJailbreakSkill (S2)100.00.97262.50.65975.80.91351.40.833
HarmBenchRedAgent80.50.73426.80.32766.80.62847.00.447
HarmBenchJailbreakSkill (S2)99.80.90141.00.65671.30.81748.00.612

ASR is reported as a percentage. HS denotes Harmful Score on a 0–1 scale. Higher values are better for both metrics.

References

[1] AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration

[2] AutoRISE: Agent-Driven Strategy Evolution for Red-Teaming Large Language Models

[3] RedAgent: An Autonomous Agent for Context-Aware Red Teaming of LLM Jailbreaks

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.