acceptodds
Under review as a conference paper at ICLR 2027

RT-Agent: Automated Extraction, Generation, and Adaptation of Jailbreak Attacks for LLM Guardrails

13 views

Abstract

New jailbreak techniques appear in papers, repositories, and forums faster than deployed guardrails are retrained against them. Every round costs an engineer who reads the description, reimplements the attack, builds an evaluation corpus, retrains the classifier, and checks it for regressions. We propose RT-Agent, a system that automates that path. The pipeline scrapes candidate descriptions, has a frontier LLM write an executable attack generator for each promising one and validate it against its source, scores the generated attacks against a heterogeneous ensemble of four production guardrails, fine-tunes the guard on the generated attacks, and repeats the cycle. Run over scraped documents, it promoted 30 candidates and turned seven of them into working attacks with no human in the loop, alongside six strategies reproduced from curated descriptions. The synthesized attacks raise the strict guard-jailbreak rate by up to percentage points. Fine-tuning WildGuard on seven of them in sequence lowers the share of harmful attack prompts it lets through from to on average (), suppresses each attack at its own step and partly before it, and keeps the locality on par with the base guard on standard safety benchmarks. The benign parts of the source code and data are published https://anonymous.4open.science/r/rt_agent_code-8533.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Guardrails, Adaptation, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_V2FgOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

RT-Agent is a pipeline to use public jailbreak information for guardrail updates. It selects candidate methods and uses a coding agent to implement attack generators. Afterwards, it evaluates the generated prompts and fine-tunes guards on the generated data. The training recipe consists of queries from generators and other safety training data. The experiments demonstrate that guard trained by sequential attack training data improves a lot.

Critical strengths

It addresses a real problem that guardrail need maintenance.

Critical weaknesses
  1. Paper shows a long process to deal with that problem, but it does not isolate the value of component contributions. Table 5 demonstrates gains from individual discovered strategies. However, Table 4 does not compare feedback-based selection with a fixed order on the same guard and candidate pool under matched data and training budgets. The loop demonstrates that feedback can change the training sequence, but does not quantify its benefit over a fixed schedule. This limits the empirical insight into a central feature of the pipeline.
  2. The paper is difficult to follow and writing is awkward. The paper should explain more clearly how method descriptions, attack examples, executable generators, and generated datasets relate, and how the data used for strategy selection, guard training, and evaluation differ. Although the relevant details can be recovered from the text, their presentation makes it unnecessarily difficult to understand the method and connect the experiments to the claimed contributions. As a minor editing issue, spacing around slashes in Figure 1 is also inconsistent.
Questions for authors
  1. How does feedback-based selection compare with a fixed training order?

  2. What motivates the strategy composition and fixed order in Table 3?

  3. How does sequential adaptation compare with joint training on the same data? Clarify the benefits and trade-offs of the sequential design.

3 comments

We thank the reviewer for their time and effort. We ran the comparison the reviewer asked for in W1 and Q1: the closed loop against fixed schedules on the same guard and candidate pool, with compute accounted at every step. We also compared sequential adaptation with joint training on the same data (Q3), explain the chain of Table 3 (Q2), and revised the presentation (W2).

W1 / Q1. Feedback-based selection against a fixed order.

We compared the closed loop with fixed schedules on the same guard (Llama-Guard-3-1B), the same pool of seven Discovery candidates, and the same training recipe. Compute is the training FLOPs of all steps plus the FLOPs of every gate run, which are charged to the loop, in PFLOP (101510^{15} FLOPs). M1 and M2 are measured as in Section 4.5, on the held-out prompts of each attack. The schedules train on different subsets of the pool, so we average them over the same attacks for every schedule: M1 pool and M2 pool are means over the seven Discovery attacks D1-D7, the pool the gate chooses from, whether or not the schedule trained on them; M1 untrained and M2 untrained are means over the six curated attacks (ReNeLLM, BitBypass, LogiBreak, RespAttack, FITD, Humanize), which no schedule trains on.

ScheduleOrderM1 poolM1 untrainedM2 poolM2 untrainedCompute (PFLOP)Runs
Base guard—49.732.268.577.3——
Closed loopranked7.111.670.882.1323 seeds
Fixed, stopped at the loop's budgetranked17.015.176.084.7253 seeds
Fixed, all seven attacksranked9.618.891.292.2693 seeds
Closed looprandom5.012.968.081.7365 orders
Fixed, stopped at the loop's budgetrandom21.922.478.090.2305 orders
Fixed, all seven attacksrandom8.917.987.791.5695 orders
Gate applied once to the base guardranked10.317.876.590.1363 seeds

"Stopped at the loop's budget" is the fixed schedule after its last step that fits the mean compute of the loop. M1: harmful bypass, lower is better; M2: acceptance of wrapped benign prompts, higher is better

The closed loop reaches the lowest M1 of all schedules in the table. On the untrained attacks both loop rows are below every fixed schedule: 11.6% and 12.9% against 15.1% or higher. The same holds on the pool. With the ranked order the loop is ahead of the full fixed schedule in each of the three seeds, on both sets. At matched compute, a fixed order is far behind, and it does not catch up even when it trains on all seven attacks. At the compute of the loop it is at 17.0% on the pool and 15.1% on the untrained attacks (21.9% and 22.4% with random orders). Trained to the end, at 2.2 times the compute of the loop, it reaches 9.6% and 18.8%, which is still 2.5 and 7.2 points above the loop.

The loop approach is better because after each step the gate is re-run against the adapted guard, and the next candidate that still passes it is trained on; the loop stops when none passes. It trains on two to four of the seven candidates (D2 and D3 in every run) and spends no steps on attacks the guard already blocks. Applying the gate once to the base guard is not enough: it gives 10.3% and 17.8%.

The ranked order also helps: stopped at the loop's budget, a fixed schedule is well ahead with the ranked order (15.1% against 22.4% on the untrained attacks), because the ranking puts stronger candidates first. Once all seven attacks are trained on, the order no longer matters.

Moreover, the loop does not lower M2, it leaves the acceptance of wrapped benign prompts at the level of the base guard or above, on the pool and on the untrained attacks. The fixed schedule over all seven attacks raises it to 91.2% on the pool, because it also trains on the wrappers of the attacks the guard already blocks.

Q2. Composition and order of the chain in Table 3.

Both are arbitrary. Table 3 does not test selection, it tests whether sequential adaptation holds over many steps. The chain consists of seven of the thirteen attacks in an arbitrary order, and the other six are held out to measure transfer. Selection by the gate is tested separately, on Llama-Guard (Table 4) and in the comparison above (W1 / Q1).

The order has little effect on the outcome: in the comparison above, random orders of the same seven attacks end at the same M1 as the ranked order once all seven are trained on (8.9% against 9.6% on the pool). In Table 3 itself, every attack of the chain ends at or below 4.1%, whichever step it was trained at.

W2. How descriptions, examples, generators and datasets relate, and how the data roles differ.

We revised the presentation of the method around both questions.

  • Figure 1. Every arrow between stages now names the data it carries; a scraped document yields a method description and one ready-to-use attack prompt (Stage 2); a coding agent turns the description into an executable generator (Stage 4); the generator wraps vanilla harmful queries into an attack dataset (Stage 5). The figure also shows the split into Curated and Discovery modes, the ASR gate as a reduced run of Stages 5-7, and the loop that returns the adapted guard to the gate.
  • Section 2 is reordered so that terms are introduced before they are used. It now defines the metrics (strict guard-jailbreak rate, M1, M2) before the stages, and the ASR gate has its own paragraph between Stages 4 and 5. Stages 2, 4 and 5 now state what the extractor and the generators produce.
  • We also made naming consistent across sections and fixed smaller issues (all the changes are highlighted in blue\color{blue}{\text{blue}}).

The data play three separate roles:

  • Strategy selection happens only at the ASR gate (Section 2.1, Table 9): AdvBench queries, the first 100 and then all 520, wrapped by the candidate generator and scored against the gating guard. No guard is trained at this point.
  • Guard training (Section 3, Training data): per attack, up to 1,000 of its harmful prompts and safe prompts at 2:1 (benign WildGuardMix-train queries wrapped by the same generator), mixed with 1.5x as many WildGuardMix-train rows.
  • Evaluation: up to 800 harmful prompts per attack, held out at the level of the vanilla query before any pool is built, so no query of an attack's training pool returns in that attack's held-out set under any wrapper (M1); 197 benign WildGuardMix-test prompts wrapped by each generator (M2, Appendix A); and five external benchmarks (Section 3).

Because the gate sees only AdvBench, Table 13 breaks the lifts down by dataset, and they hold on the four datasets the gate never saw. For example, D1 against Granite-Guardian gains +89.6 pp on AdvBench and +72.0 / +82.0 pp on HarmBench / JBB-Behaviors.

Q3. Sequential adaptation against joint training on the same data.

Joint training ends lower at the same compute (same metrics and units as in W1 / Q1):

TrainingAttacksM1 poolM1 untrainedM2 poolM2 untrainedCompute (PFLOP)
Sequentialall seven, ranked order9.618.891.292.269
Jointall seven4.713.186.789.669
SequentialD2, D1, D3, D6 (Table 4)8.217.375.187.133
JointD2, D1, D3, D63.59.872.182.133

On the same data and compute, one joint pass leaves M1 pool at 4.7% against 9.6% for the sequential chain, and at 3.5% against 8.2% on the four attacks of Table 4 (the two joint runs and the four-attack chain are single runs). This is because a sequential step partly undoes earlier ones, which Section 4.3 reports as partial forgetting. The joint guard is also stricter on benign prompts: its M2 is 3 to 5 points lower than that of the sequential chain, so part of its gain in M1 comes from a stricter guard.

We also emphasize that the sequential design is motivated by the problem setup: we consider a scenario of continuous Internet screening to detect and adapt to new attacks as soon as possible, since if we wait for a certain pool of attacks to accumulate for fine-tuning, malicious actors will have the opportunity to bypass the current guardrails. Moreover, adding the seventh attack to the chain costs 10 PFLOP, while retraining jointly on all seven costs 69. Retraining jointly at each of the seven arrivals costs 315 PFLOP in total, against 69 for the chain. At the same time, part of the advantage of joint training rests on the loop's selection. That selection is an outcome of the sequential loop: which candidates still pass the gate depends on the steps already taken. The best joint result uses the four attacks the loop selected (3.5% and 9.8%, against 7.1% and 11.6% for the loop itself). Without it, a joint pass over all seven candidates costs twice the compute of the loop and is no better on the untrained attacks (13.1% against 11.6%), although it is lower on the pool (4.7% against 7.1%).

If we have addressed all of your questions, the authors would be very grateful if the esteemed reviewer would reconsider the score.

Thank you for the revision and experiments. The closed loop lowers both harmful bypass and benign acceptance compared with fixed schedules. And joint training shows the same pattern compared with sequential training. I will raise my score from 1 to 2.

Reviewer_K67SOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes RT-Agent, an automated attack discovery and adaptation framework for Large Language Model safety guardrails. This framework can extract jailbreak strategies from sources such as academic papers, code repositories, and forums, automatically generate and filter attack programs, and subsequently construct training data to fine-tune safety guardrails. Experiments demonstrate that RT-Agent can reproduce and discover various effective attacks, and enhance the guardrail’s defense capabilities against novel jailbreak attacks while maintaining a high normal request pass rate and standard safety benchmark performance.

Critical strengths

1、Novel research perspective. The paper does not solely focus on attack generation or guardrail fine-tuning; instead, it integrates attack method extraction and the adaptive updating of guardrails into a unified pipeline. This end-to-end design provides a relatively novel research approach for continuously discovering new attacks and updating safety guardrails. 2、Comprehensive experimental setup with sufficient reproducibility details. The paper provides detailed data construction methods, training configurations, filtering rules, and supplementary experiments, which helps readers understand and reproduce the proposed system. 3、Clear research motivation with practical significance. The paper clearly points out the efficiency and scalability limitations of manual collection, reproduction of attacks, and guardrail updating. The overall research problem is well-defined and holds strong practical application value.

Critical weaknesses

1、Insufficient baselines for comparison. The paper only compares with Rapid Response, a single guardrail fine-tuning method, despite introducing Rapid Response, AdaptiveGuard, SESG, and DARWIN in the Introduction. The current results are insufficient to fully demonstrate the advantages of the proposed method over other guardrail adaptation methods. It is recommended to include more relevant baselines.

2、Limited validation scope for attack reproduction capabilities. The CURATED MODE (Table 1) includes only five published attack methods, while “Humanize” is a baseline constructed by the authors. The number of existing attacks is small, and all originate from 2024–2025, with no attack methods from 2026 included in this mode. Therefore, the current experiments provide limited support for the system’s ability to broadly reproduce different types of and recent attack methods.

3、Lack of ablation studies on key training designs. During guardrail fine-tuning, the paper simultaneously employs normal samples wrapped by the attack generator, a mixture of existing safety data, and KL constraints, but it does not evaluate the independent contributions of these designs respectively. Consequently, it is difficult to determine which components are responsible for the improvement in defense capabilities, the maintenance of the normal request pass rate, and the mitigation of catastrophic forgetting.

Questions for authors

1、Supplement sufficient baselines to demonstrate the effectiveness of the proposed guardrail fine-tuning method.

2、Include more jailbreak attack methods, especially the most recent ones, to validate the attack reproduction capability.

3、Complete the ablation experiments to analyze the contribution of different training methods.

3 comments

We thank the reviewer for the insightful feedback! In response to your three points, we added AdaptiveGuard as a second adaptation baseline (W1 / Q1), ran ten attacks published in 2026 through the pipeline and adapted Llama-Guard to the two strongest (W2 / Q2), and ablated the three components of the training recipe (W3 / Q3).

W1 / Q1. The revision adds AdaptiveGuard (Yang et al., ASE 2025) as a second adaptation baseline next to Rapid Response (Table 5). Each method adapts WildGuard independently to D2, D4 and BitBypass and is scored on the same held-out prompts (M1: harmful bypass, lower is better; M2: acceptance of wrapped benign prompts, higher is better).

MethodM1 ownM2 ownM1 otherM2 otherΔF1 (5 benchmarks)
Base guard14.4–37.991.9–94.920.392.3—
Rapid Response0.00.08.1–12.579.8–87.9−0.034 to −0.048
AdaptiveGuard0.00.00.3–2.743.4–66.4−0.025 to −0.044
RT-Agent1.6–4.891.9–92.410.0–12.890.9–93.0+0.002

The methods sit at different points of the M1/M2 trade-off. AdaptiveGuard, updated online on 50 attack prompts in under a minute, blocks the trained attack completely, but it also rejects all of its wrapped benign prompts, accepts 26–49 points fewer of them on the six other attacks, and loses 0.025–0.044 mean F1. Its update sees harmful prompts only; giving it all 580–850 harmful prompts of our pool deepens this (M2 other 35.8–47.5%, Appendix A). Rapid Response likewise reaches M1 = M2 = 0% on the trained wrapper and loses 0.034–0.048 mean F1, three to four times the ±0.011 bootstrap half-width. RT-Agent keeps 1.6–4.8% bypass on the trained attack but accepts 91.9–92.4% of its wrapped benign prompts and leaves F1 within noise.

SESG and DARWIN cannot be run in this setting:

  • SESG (arXiv 2608.08471, Aug 2026) releases only its test sets; the pipeline, guard and training data are not public. Its own adaptive baseline is AdaptiveGuard, which we now include.
  • DARWIN (arXiv 2607.19829, Jul 2026, revised Sep 2026) releases code and a guard checkpoint, but the release does not include everything needed to rerun its loop on our attacks. It also targets a different task: co-evolving an attacker and a guard rather than adapting to a newly observed attack, so scoring its checkpoint would compare base guards, not adaptation methods.

W2 / Q2. We took a set of 2026 jailbreak papers and ran the pipeline from generation to fine-tuning. The table gives the strict guard-jailbreak rate (%) on the 520 AdvBench prompts; bold marks rates above the gate threshold of 5%.

AttackarXivLlama-GuardWildGuardQwen3-GuardGranite-Guardian
IICL2609.0837360.20.00.61.5
Intent Laundering2602.1672956.22.55.21.9
S2C2603.161921.21.50.412.1
SpatialJB2601.093211.70.00.00.2
CodeMimicry2609.399021.20.00.02.1
Prompt Overflow2605.231960.20.20.00.2
VAR2606.044830.00.20.00.0
Posterior Attack2606.056140.00.00.00.0
RoguePrompt2607.273730.00.00.00.0
JailbreakOPT2606.114257.51.70.22.7

IICL and Intent Laundering pass Llama-Guard on 60.2% and 56.2% of the prompts. Intent Laundering also crosses the threshold on Qwen3-Guard, and S2C on Granite-Guardian. No attack crosses it on WildGuard. The remaining 6 attacks do not pass the gate. However, they are fully reproduced by our pipeline and show our pipeline's ability to reproduce attacks of different mechanism families.

We then fine-tuned the Llama-Guard in accordance with our pipeline to the new attacks. All the protocols remain the same as in the original paper

AttackM1 own, base → adaptedM2 own, base → adaptedF1, base → adapted
IICL86.6 → 5.893.9 → 94.90.728 → 0.778
Intent Laundering78.2 → 2.697.0 → 93.40.728 → 0.763

M1 falls from 86.6% to 5.8% and from 78.2% to 2.6%, M2 stays within 3.6 points of the base guard, and mean F1 rises by 0.050 and 0.035.

W3 / Q3. Ablations of the training recipe.

We ran leave-one-out ablations on the Llama-Guard chain of Table 4, with the attack order fixed to the one the loop selected (D2, D1, D3, D6), so variants differ only in the dropped component. After step 4 we have:

VariantM1 trainedM1 untrainedM2 trainedM2 untrainedF1
Base guard72.029.977.061.00.728
Full recipe (Table 4)9.77.883.961.50.787
w/o wrapped benign0.07.49.560.50.780
w/o WildGuardMix14.08.280.151.30.764
w/o KL8.88.484.162.10.787

The outcomes are as follows:

  • Defense comes from the attack pool itself: every variant cuts M1 from 72.0 to at most 14.0 on trained attacks and from 29.9 to at most 8.4 on untrained ones.
  • Wrapped benign prompts keep normal requests passing; without them the guard learns the wrapper instead of the harm: M1 on trained attacks drops to 0.0, but it rejects 90.5% of benign prompts in the same wrappers (M2 9.5 vs. 83.9), already at step 1 (52.4 vs. 74.7).
  • The WildGuardMix mixture limits forgetting: without it, F1 is lower at every step, by at least 0.020, and M2 on untrained wrappers falls below the base guard (51.3 vs. 61.0).
  • KL has relatively small effect at the chain lengths we ran: over the four Llama-Guard steps, removing it moves M1 by at most 1.3 pp, M2 by at most 1.7 pp and F1 by at most 0.001; over the seven steps of the WildGuard chain (Table 3), by at most 0.7 pp, 0.5 pp and 0.001. The term always anchors to the original base guard, and the drift from that guard accumulates with each sequential fine-tune: on Llama Guard, the KL to the base on WildGuardMix prompts roughly doubles from step 1 to step 4. The term may therefore matter in longer chains.

If we have addressed all of your questions, the authors would be very grateful if the esteemed reviewer would kindly reconsider the score.

Thank you for the detailed response. While your experiments address some of my concerns, I maintain my original score.

Reviewer_HVhxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposed RT-Agent, a system that automate the adaptation of new jailbreak techniques. The proposed pipeline includes several stages where the first half search for, extract, and generate attack and the second half score them against guard and improve the guard on the new jailbreaks.

Critical strengths
  • S1: The work has a significant engineering contribution on the end-to-end auto jailbreak adaptation. Although the individual components are established techniques from prior works, the integration address the practical problem with a comprehensive experiment result.

  • S2: Adaptive attack selection optimize the updated guardrail against the all existing attacks, which is a practical feature for production-wise deployment.

Critical weaknesses
  • W1: The framework deduplicates attacks based on the similarity of extracted examples rather than the underlying mechanisms. It may merge distinct jailbreak methods that share similar wording, without considering the differences between jailbreak types.
Questions for authors
  • How does the system tell if the jailbreak attack is a different variation of the existing version or the new type of the attack?

  • After fine-tuning in Stage 9, does evaluation test new variations of the same jailbreak method, or only new requests using the same attack template? Is this tested in Stage 10?

3 comments

We thank the reviewer for the insightful feedback! In response to your points, we improved the deduplication stage, it now compares attack mechanisms and repeats the audit of the paper on it (W1), described how the new stage tells a variation from a new attack type (Q1), and clarified what is evaluated after fine-tuning (Q2). All the changes are highlighted in blue\color{blue}{\text{blue}} in the revised manuscript.

W1. Deduplication by attack mechanism.

The revision improves the deduplication stage (Appendix N). The initial approach compared one extracted example per document, and the examples were merged by topic rather than by jailbreak tactic. The unit of deduplication is now an attack rather than a document.

  • Attack extraction is done by a model (DeepSeek-V4-Flash-0731) that reads the whole document and lists every attack it describes, so a document with n attacks yields n records. In the same call it drops attacks whose goal is not to obtain refused content, such as data leakage or attacks on agent tools. Only attacks whose name or quote occurs in the document are kept.
  • A construction recipe is written for each attack, from which the attack can be rebuilt without the document. It contains the verbatim template, the exact transformation of the request, the order of turns and what is optimised. The model also copies a ready attack prompt from the document when there is one.
  • Similarity is computed from the recipe alone, which is embedded by Qwen3-Embedding-0.6B with an instruction to capture the template and the transformations and to ignore the topic. The similarity of two attacks is the cosine of their vectors, minus 0.12 if both come from the same source, because tools and benchmarks describe many of their attacks with the same context.
  • Clustering uses average linkage, the same algorithm as in the paper, at a cut of 0.85, and each cluster keeps its medoid.

We repeated the audit of the paper on the new stage with the same judge (DeepSeek-V4-Flash-0731) and the same protocol. The check uses 100 pairs. False removals are measured on 60 randomly drawn removed attacks, each judged against the representative of its cluster. For missed duplicates we take the attacks closest to each other that the stage left apart, 40 randomly drawn pairs from different sources with similarity within 0.05 below the cut, and ask whether they are the same method.

Submitted stageNew stage
Unit of deduplicationdocumentattack
What is comparedone extracted exampleconstruction recipe
Input2,915 documents7,014 attacks from 2,437 documents
Kept1,942 documents6,229 attacks from 2,302 documents
False removals (%)65.48.3
Missed duplicates near the cut (%)2.732.5

False removals fall from 65.4% to 8.3%. Between independent sources the judge rejected 4 removals out of 52, and the remaining errors come from attacks listed by the same tool (1 out of 8).

The cut and the penalty trade the two errors against each other. The table uses average linkage and 100 new pairs per row, with 95% half-widths of 6 to 15 percentage points.

CutSame-source penaltyKept attacksFalse removals (%)Missed duplicates near the cut (%)
0.780.124,37731.715.0
0.820.085,22820.027.5
0.850.085,97511.750.0
0.850.126,2298.332.5
0.850.156,29913.342.5
0.880.086,4675.065.0

We chose 0.85 because false removals are the costlier error. At 0.82 they are 20.0%. At 0.88 they are about the same as at 0.85 (5.0%), but 65% of the close pairs the stage leaves apart are the same method. At the cut of 0.85 the three penalties differ within their intervals.

The new stage leaves more duplicates apart than the submitted one (32.5% against 2.7% near the cut). This rate is measured only on the closest pairs the stage left apart, not on all kept attacks. The comparison is not entirely equal, since the unit is now an attack and one popular attack occurs in dozens of sources. Moreover, the two errors cost differently. A false removal loses an attack for good, because its generator is never built and the guard never sees it. A missed duplicate only leaves a second copy of a known attack among the candidates. In the closed loop the ASR gate is re-run against the adapted guard, and attacks the previous adaptation has absorbed no longer pass it (Section 4.5), so such a copy costs one generator build and one 100-prompt gate run. We therefore chose a cut with fewer false removals and more remaining duplicates.

We also checked that the new stage does not break the rest of the pipeline (Appendix N). The generator now receives the text of the document together with a pointer to the specific attack, and the ranking is still based on the length of the extracted example. We ran one step of the loop of our algorithm, obtained a dataset of harmful requests and then fine-tuned the guard to defend against it. Of the 4 top-ranked tactics, 4 gave a working generator and 1 passed the gate, and after fine-tuning Llama-Guard, M1 on the selected attack (a Tyler Durden persona jailbreak) fell from 43.6% to 12.8% at M2 89.3% and F1 0.783.

Q1. Telling a variation from a new attack type.

An attack is treated as a variation when its recipe is, on average, more similar to the recipes of an existing cluster than the cut of 0.85. The similarity is averaged over all attacks of the cluster, so one accidentally similar member is not enough. The attack is then merged, and only the medoid of the cluster is kept, otherwise it is a new cluster. The table in W1 gives how often this decision is wrong in each direction.

Q2. What is evaluated after fine-tuning.

After fine-tuning, M1 is measured on test harmful prompts, up to 800 per attack. These prompts are produced by the same generator, so they are new requests and new samples of the same method, not a different variant of it. The guard is also scored on attacks it was never trained on (M1 untrained). In the closed loop on Llama-Guard, M1 on the six untrained attacks falls from 32.2% to 11.6% (please see our response to Reviewer V2Fg). Stage 10 tests neither of the two. It checks regression on five external safety benchmarks.

If we have addressed all of your questions, the authors would be very grateful if the esteemed reviewer would kindly reconsider the score.

Thank you for addressing my comments and questions. I will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.