- Rating
- 2
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2(−1)
- Soundness
- 2(−1)
Read review
Paper summary
JailbreakSkill is essentially a red-teaming method with lifelong learning built around skills, enabling the evolution of those skills to produce stronger jailbreak attacks against LLMs. Authors use rewriting, failure analysis, and evolution to expand and improve their attack skill library. The choice of the next skill is guided by a risk-conditioned UCB score.
Evaluations cover AdvBench and HarmBench, using an ASR evaluation rubric adapted from X-Teaming [1]. The main text includes evaluations such as:
- ASR@1, ASR@10, and ASR@30.
- Coverage across skill categories and metrics on the contributions of failure-driven skill evolution.
- Attack transfer success rates across models and held-out prompts.
References:
- X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents, Salman Rahman et al.
Critical strengths
Strengths of the method include:
- Comprehensive experimental analysis, including transfer capabilities and evaluations across multiple model families, including GLM 5.2 (in the appendix).
- Using a UCB score to guide the choice of the next strategy seems like a sound idea compared to random selection.
- The strategies are diverse and drawn from multiple popular works.
- Target-model query budgets are explicitly stated for JailbreakSkill.
- The skill framing is useful but has been discussed in prior work, as mentioned by the authors themselves.
Critical weaknesses
Despite these strengths, the weaknesses are substantial:
- My biggest question is how a skill in this paper differs from a strategy used by popular methods such as AutoDAN-Turbo [1], AutoRedTeamer [2], X-Teaming [3], and PLAGUE [4]. An evaluation demonstrating whether the skill format yields substantial improvements over a standard strategy library is missing. For LLM-based rewriting, invoking a skill still ultimately appends instructions to the start of a user prompt to the LLM. The benefit of the additional format and interface needs to be demonstrated.
- I am very confused about the interpretation of ASR@10 and how it relates to independent evaluations of the same method. The paper defines ASR@10 using a target-model query budget, which differs from a Pass@K-style evaluation over independent runs. Stage 1 keeps the skill definitions fixed, but updates selection heuristics across attack steps. Stage 2 adds 20 more target-model calls, in which unresolved behaviors can be drawn from a pool of evolved skills that persist across categories. These results, therefore, measure an adaptive procedure rather than independent runs from the same initial state. The authors need to distinguish these settings clearly and explain whether the baselines receive comparable opportunities to learn and reuse experience across evaluations.
- The benefits of skill evolution in Figure 4, appear to be negligible compared with simply spending the additional budget on the initial library in three of the four settings. The exception is GPT-5.4 on AdvBench, where evolution produces a substantial improvement. The contribution of evolution, therefore, needs to be assessed carefully, separately from additional queries and from the skill format itself.
- The baseline budget accounting needs clarification. JailbreakSkill uses a Stage 1 budget of 10 target-model calls, whereas AutoDAN-Turbo lists max_iteration=6, alongside separate warm-up and lifelong-learning parameters. However, its reported average target-call count exceeds six, so this parameter cannot represent its total budget. The authors need to explain how these parameters map to the shared query budget and establish that the methods are compared under like-for-like settings.
- Faithfulness to the original goal can also be embedded into the StrongReject evaluation [4]. Although the ASR judge considers original-goal fulfillment, the StrongREJECT-style harmfulness results [6] can diverge substantially from ASR. The evaluation should explicitly distinguish harmful content from fulfillment of the original objective, similar to PLAGUE [4].
- A comparison against RedAgent [5] is missing. A public implementation seems to be available, and a comparison against this baseline is warranted.
References
- AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs, Xiaogeng Liu et al.
- AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration, Andy Zhou et al.
- X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents, Salman Rahman et al.
- PLAGUE: Plug-and-play Framework for Lifelong Adaptive Generation of Multi-turn Exploits, Neeladri Bhuiya et al.
- RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent, Huiyu Xu et al.
- A StrongREJECT for Empty Jailbreaks, Alexandra Souly et al.
Questions for authors
- Clarify the budgets and evaluate the methods under the same target-model query budgets? Please distinguish target calls, internal attack steps, and independent runs when explaining ASR@1, ASR@10, and ASR@30. State this clearly for all methods and baselines being compared, and explicitly mention which behaviors persist across turns. If learning across evaluated behaviors is part of the algorithm, applicable baselines should be evaluated under the same settings.
- Add results for RedAgent [1], or explain specifically why its public implementation cannot support a faithful comparison?
- Can the authors show the effect of representing and evolving the same procedures as skills versus a standard strategy library expressed in general text? Keep everything else fixed.
- Please explain Figure 4 clearly. Evolution performs similarly to additional sampling from the initial library in three settings, while GPT-5.4/AdvBench shows a substantial gain?
- Can the authors validate the StrongREJECT evaluation prompt [2] against fulfillment of the original goal and report goal faithfulness separately as is done in PLAGUE [3]?
References:
- RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent, Huiyu Xu et al.
- A StrongREJECT for Empty Jailbreaks, Alexandra Souly et al.
- PLAGUE: Plug-and-play Framework for Lifelong Adaptive Generation of Multi-turn Exploits, Neeladri Bhuiya et al.
12 comments
Rebuttal by Authors
Weakness 1.2 and Question 3: Skill representation versus a standard strategy library
We thank the reviewer. We provide a controlled representation comparison for two fixed mechanisms: Base64 encoding (Arms A/B) and role-play persona rewriting (Arms C/D). Within each pair, the mechanism and seeds are fixed, and the same executor is used when model interpretation is required. The C/D effectiveness supplement uses the same GPT-4.1 target, GPT-4o judge, and one-query budget for both arms.
We accept the reviewer's core observation regarding LLM-rewrite skills. For llm_rewrite skills, the target-facing artifact does reduce to a strategy prompt prepended to the seed. We now state this explicitly in §4.2.
What a skill provides beyond a strategy. A strategy in AutoDAN-Turbo, AutoRedTeamer, X-Teaming, or PLAGUE is natural-language guidance consumed by an attacker LLM as context. A JailbreakSkill skill provides three things a strategy does not: (1) an executable contract — a fixed JSON-over-stdio interface that holds deterministic Python templates, LLM-rewrite instructions, or hybrids behind the same interface; (2) an admission gate — every candidate passes schema validity, seed-intent preservation, and local executability checks before library entry; and (3) addressability for evolution — each skill carries a stable name, category, stage, and execution mode, which is the identity that Eq. 1 accumulates per skill × risk category, enabling in-place refinement without touching the rest of the library.
Controlled comparison on 30 AdvBench seeds, same executor, same seeds, with only the representation varied (Arm A/B: Base64 encoding; Arm C/D: role-play persona rewriting):
Table R2. Skill representation versus strategy prose under otherwise identical settings.
| Arm | Representation | Valid B64 ↑ | ASR on GPT-4.1 ↑ | HS ↑ | Output length (std) |
|---|---|---|---|---|---|
| A: Skill run.py (deterministic) | Exact Python execution | 1.000 | — | — | 15.0 (0.0) |
| B: Strategy prose | Executor reconstructs encoding | 0.933 | — | — | 5.7 (5.7) |
| C: Skill run.py (LLM-rewrite) | Structured skill instructions | — | 23.3 (7/30) | 0.409 | 92.9 (11.6) |
| D: Strategy prose | Flat natural-language description | — | 6.7 (2/30) | 0.168 | 39.1 (7.5) |
Arm A uses zero executor calls (deterministic); Arms B, C, and D each use one call per seed. Valid B64 applies only to the Base64 pair (A/B), and ASR/HS only to the role-play pair (C/D); non-applicable cells are marked "—".
Arm A vs. B — deterministic skills cannot be faithfully represented as strategies. Arm A calls base64.b64encode() directly: 100% valid Base64, zero executor calls, zero output variance. Arm B asks the executor to reconstruct the encoding from prose: a 6.7% failure rate, and 0% of outputs match Arm A — even when the executor produces valid Base64, it encodes different content than the seed. The cost is visible at scale: Table 10 reports 3.08 attack/rewrite calls per seed (Stage 1) for JailbreakSkill versus 17.58 for AutoDAN-Turbo and 9.40 for MemoAttack, with AQC 4.39 versus 8.19 and 7.49, because deterministic skills require no model inference.
Arm C vs. D — attack effectiveness of the frozen role-play outputs. We additionally sent each saved prompt to GPT-4.1 once (one target-model query per prompt) and scored the response with GPT-4o against the original seed, success defined as score 5; HS uses the StrongREJECT rubric. Over the 30 AdvBench seeds — one biological-harm seed is counted as a failure for both arms — the skill representation (Arm C) reaches 23.3% ASR (7/30) and 0.409 HS, versus 6.7% (2/30) and 0.168 for the strategy prose (Arm D). Both scores use GPT-4o-2024-08-06 with the repository's native scoring templates. This is a small, single-mechanism comparison: the paired ASR difference has an exact two-sided McNemar p-value of 0.125, so it indicates a consistent direction rather than a statistically significant advantage, and it does not isolate the effect of evolution. It does, however, show that the higher-fidelity, structured skill output is at least as effective as the prose version, so the format's benefits do not come at the cost of attack strength.
Why skill-based evolution matters. Among all 1,933 unique evolved skills accumulated across our runs, 798 (41%) are deterministic_template and 450 (23%) are hybrid — execution modes that a strategy library cannot represent or validate. A strategy library retains candidates as text descriptions that the executor must re-interpret at every invocation, with the same fidelity gap as Arm B applying to every call.
Summary. The skill format contributes in two ways: (a) deterministic and hybrid execution with 100% fidelity and zero executor calls (Arm A vs. B); and (b) an admission-gated, addressable unit for evolution that yields 64% of evolved skills in modes a strategy library cannot represent.
Rebuttal by Authors
Weakness 2 and Question 1: What ASR@K measures, and comparability with baselines
We thank the reviewer for the opportunity to clarify the evaluation protocol. We address the budget definition, the distinction between target calls and internal steps, and cross-behavior learning together, since they are aspects of the same concern.
What ASR@K measures. Our main experiments evaluate adaptive attack performance under a fixed target-model query budget — not Pass@K over independent, stateless runs. For a given seed behavior, a budget of K target-model calls means the method may make up to K attempts, stopping at the first success. ASR@K is the fraction of seeds solved within that budget. This is the standard protocol in automated red-teaming, and we will state it explicitly in §5.2.
We distinguish three quantities that ASR@K does and does not conflate:
- Target-model calls (the budget). Calls to the model under attack. This is the scarce, shared resource. All methods receive the same maximum (10 for ASR@10, 30 for ASR@30) and stop after the first success.
- Internal attack steps. Per-attempt computation that does not consume the target budget — for example, TAP's tree expansion and pruning, ReNeLLM's rewrite iterations, AutoDAN-Turbo's strategy retrieval, or JailbreakSkill's UCB ranking. These differ across methods but do not count against the shared budget, as reported in Table 10 (Appendix C.5).
- Independent runs (Pass@K). K stateless draws from the same distribution, with no adaptation. Our protocol is not this. We will add an explicit statement distinguishing the two settings in §5.2.
How each method uses the budget, and what persists across turns. All baselines operate under the same target-query limit and evaluation criterion. They fall into two groups (Table R3):
Table R3. Within-behavior adaptation and cross-behavior reuse in each method.
| Method | Intra-behavior adaptation | Cross-behavior reuse |
|---|---|---|
| PAIR | Iterative prompt refinement | — |
| TAP | Tree search over refined prompts | — |
| ReNeLLM | Iterative prompt rewriting | — |
| AutoDAN-Turbo | Strategy retrieval from attack history | Lifelong strategy accumulation across seeds |
| MemoAttack | Tree search with memory-guided generation | Cross-seed memory reuse |
| JailbreakSkill (S1) | UCB skill selection updated from outcomes | Risk-conditioned statistics shared across seeds |
| JailbreakSkill (S2) | Evolved skills evaluated on unresolved behaviors | Validated skills added to the library for subsequent categories |
PAIR, TAP, and ReNeLLM adapt within a single seed but do not carry experience across seeds. AutoDAN-Turbo, MemoAttack, and JailbreakSkill additionally reuse experience across seed prompts within a single run. This reflects each method's own design rather than an asymmetry in our evaluation: all methods are evaluated under the same protocol, and the cross-seed-reuse group is represented by baselines (AutoDAN-Turbo, MemoAttack) that do exactly the same thing. No baseline is restarted from a stateless initial state between seed prompts: those that accumulate memory do so by design, and those that do not (PAIR, TAP, ReNeLLM) start each seed fresh by their own design.
On the relationship to Pass@K. The reviewer is right that ASR@K under an adaptive protocol measures something different from Pass@K over independent runs. The former measures the ceiling an adaptive method can reach under a fixed interaction budget; the latter measures the marginal success probability of a single attempt. Both are legitimate, but they answer different questions. For deployment-oriented safety evaluation — where an attacker has a fixed query budget and can observe model responses — the adaptive protocol is the operationally relevant one. We will state this distinction explicitly.
On ASR@30 combining Stage 1 and Stage 2. ASR@30 is not 30 fresh draws from a fixed pool. Stage 1 uses target calls 1–10 over the initial 16-skill library; Stage 2 uses calls 11–30, during which evolved skills (generated from the failure traces accumulated in Stage 1) are evaluated on unresolved behaviors. This is an adaptive procedure within a single 30-call budget, not independent runs. The budget-matched control in Appendix C.7 (Figure 4) isolates the contribution of evolution from the contribution of additional budget: one arm spends calls 11–30 evolving new skills, and the other spends the same calls resampling the initial library. Where the initial library still has headroom (GLM 5.2), resampling recovers most failures and evolution adds only 1–2 points. Where the library is exhausted (GPT-5.4/AdvBench), resampling saturates at 25% while evolution reaches 62.5%, a 37.1-point gap. This control confirms that the ASR@30 gain is attributable to the new mechanisms produced by evolution, not merely to additional budget.
Rebuttal by Authors
We thank the reviewer for the detailed and constructive feedback, and for acknowledging the breadth of our experimental analysis.
Weakness 1.1: How a skill differs from a strategy
We thank the reviewer for pressing on this point. We first clarify the distinction between JailbreakSkill and each of the four cited systems, then provide the controlled comparison the reviewer requested.
Table R1 summarizes how each method accumulates and applies attack knowledge.
Table R1. Knowledge representation and update mechanisms in related methods.
AutoDAN-Turbo and RedAgent accumulate experience as strategy descriptions, examples, and interaction memory, but neither provides a unified component representation that spans attack, diagnosis, and evolution. AutoDAN-Turbo retrieves strategies by matching target responses against historical experience; RedAgent combines target context, historical experience, and interaction feedback into attack plans that contain strategies and examples. JailbreakSkill, by contrast, invokes these procedures directly as executable skills through a fixed interface.
AutoRedTeamer drives library expansion primarily from external research literature, whereas JailbreakSkill uses accumulated unresolved cases to diagnose shared failures and generate new executable skills, so that internal evaluation experience directly participates in capability growth. AutoRISE evolves a single monolithic attack program, adjusting its techniques and their organization in response to evaluation feedback. Our RQ1 shows that different attack mechanisms differ substantially in effectiveness across target models and provide complementary coverage, which makes maintaining and managing diverse methods an important requirement. JailbreakSkill therefore stores each method as an independent skill, so that methods can be invoked, evaluated, and updated separately, and the library can grow through a unified interface.
X-Teaming and PLAGUE are multi-turn dialogue attack methods whose per-query budget semantics differ from ours. In a multi-turn setting with a maximum of T turns, a budget of K target-model queries supports at most ⌊K/T⌋ complete attack episodes; in our single-turn rewriting setting, K queries correspond to K independent attempts on the same seed. With T = 5, for example, ASR@10 supports at most 2 complete multi-turn episodes and ASR@30 at most 6, which places multi-turn methods at a structural disadvantage under a single-turn evaluation protocol for reasons unrelated to their actual capability. This is why prior single-turn red-teaming work (e.g., AutoDAN-Turbo, MemoAttack) does not directly compare ASR against multi-turn methods. We follow the same convention and restrict our main baseline comparison to single-turn methods; we have added a clarifying note in §5.1. At the request of Reviewer T6CT, we additionally report PLAGUE as a separate supplementary comparison, with all of its target-model calls counted against the same budget.