- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
RT-Agent is a pipeline to use public jailbreak information for guardrail updates. It selects candidate methods and uses a coding agent to implement attack generators. Afterwards, it evaluates the generated prompts and fine-tunes guards on the generated data. The training recipe consists of queries from generators and other safety training data. The experiments demonstrate that guard trained by sequential attack training data improves a lot.
Critical strengths
It addresses a real problem that guardrail need maintenance.
Critical weaknesses
- Paper shows a long process to deal with that problem, but it does not isolate the value of component contributions. Table 5 demonstrates gains from individual discovered strategies. However, Table 4 does not compare feedback-based selection with a fixed order on the same guard and candidate pool under matched data and training budgets. The loop demonstrates that feedback can change the training sequence, but does not quantify its benefit over a fixed schedule. This limits the empirical insight into a central feature of the pipeline.
- The paper is difficult to follow and writing is awkward. The paper should explain more clearly how method descriptions, attack examples, executable generators, and generated datasets relate, and how the data used for strategy selection, guard training, and evaluation differ. Although the relevant details can be recovered from the text, their presentation makes it unnecessarily difficult to understand the method and connect the experiments to the claimed contributions. As a minor editing issue, spacing around slashes in Figure 1 is also inconsistent.
Questions for authors
-
How does feedback-based selection compare with a fixed training order?
-
What motivates the strategy composition and fixed order in Table 3?
-
How does sequential adaptation compare with joint training on the same data? Clarify the benefits and trade-offs of the sequential design.
3 comments
W2. How descriptions, examples, generators and datasets relate, and how the data roles differ.
We revised the presentation of the method around both questions.
- Figure 1. Every arrow between stages now names the data it carries; a scraped document yields a method description and one ready-to-use attack prompt (Stage 2); a coding agent turns the description into an executable generator (Stage 4); the generator wraps vanilla harmful queries into an attack dataset (Stage 5). The figure also shows the split into Curated and Discovery modes, the ASR gate as a reduced run of Stages 5-7, and the loop that returns the adapted guard to the gate.
- Section 2 is reordered so that terms are introduced before they are used. It now defines the metrics (strict guard-jailbreak rate, M1, M2) before the stages, and the ASR gate has its own paragraph between Stages 4 and 5. Stages 2, 4 and 5 now state what the extractor and the generators produce.
- We also made naming consistent across sections and fixed smaller issues (all the changes are highlighted in ).
The data play three separate roles:
- Strategy selection happens only at the ASR gate (Section 2.1, Table 9): AdvBench queries, the first 100 and then all 520, wrapped by the candidate generator and scored against the gating guard. No guard is trained at this point.
- Guard training (Section 3, Training data): per attack, up to 1,000 of its harmful prompts and safe prompts at 2:1 (benign WildGuardMix-train queries wrapped by the same generator), mixed with 1.5x as many WildGuardMix-train rows.
- Evaluation: up to 800 harmful prompts per attack, held out at the level of the vanilla query before any pool is built, so no query of an attack's training pool returns in that attack's held-out set under any wrapper (M1); 197 benign WildGuardMix-test prompts wrapped by each generator (M2, Appendix A); and five external benchmarks (Section 3).
Because the gate sees only AdvBench, Table 13 breaks the lifts down by dataset, and they hold on the four datasets the gate never saw. For example, D1 against Granite-Guardian gains +89.6 pp on AdvBench and +72.0 / +82.0 pp on HarmBench / JBB-Behaviors.
Q3. Sequential adaptation against joint training on the same data.
Joint training ends lower at the same compute (same metrics and units as in W1 / Q1):
| Training | Attacks | M1 pool | M1 untrained | M2 pool | M2 untrained | Compute (PFLOP) |
|---|---|---|---|---|---|---|
| Sequential | all seven, ranked order | 9.6 | 18.8 | 91.2 | 92.2 | 69 |
| Joint | all seven | 4.7 | 13.1 | 86.7 | 89.6 | 69 |
| Sequential | D2, D1, D3, D6 (Table 4) | 8.2 | 17.3 | 75.1 | 87.1 | 33 |
| Joint | D2, D1, D3, D6 | 3.5 | 9.8 | 72.1 | 82.1 | 33 |
On the same data and compute, one joint pass leaves M1 pool at 4.7% against 9.6% for the sequential chain, and at 3.5% against 8.2% on the four attacks of Table 4 (the two joint runs and the four-attack chain are single runs). This is because a sequential step partly undoes earlier ones, which Section 4.3 reports as partial forgetting. The joint guard is also stricter on benign prompts: its M2 is 3 to 5 points lower than that of the sequential chain, so part of its gain in M1 comes from a stricter guard.
We also emphasize that the sequential design is motivated by the problem setup: we consider a scenario of continuous Internet screening to detect and adapt to new attacks as soon as possible, since if we wait for a certain pool of attacks to accumulate for fine-tuning, malicious actors will have the opportunity to bypass the current guardrails. Moreover, adding the seventh attack to the chain costs 10 PFLOP, while retraining jointly on all seven costs 69. Retraining jointly at each of the seven arrivals costs 315 PFLOP in total, against 69 for the chain. At the same time, part of the advantage of joint training rests on the loop's selection. That selection is an outcome of the sequential loop: which candidates still pass the gate depends on the steps already taken. The best joint result uses the four attacks the loop selected (3.5% and 9.8%, against 7.1% and 11.6% for the loop itself). Without it, a joint pass over all seven candidates costs twice the compute of the loop and is no better on the untrained attacks (13.1% against 11.6%), although it is lower on the pool (4.7% against 7.1%).
If we have addressed all of your questions, the authors would be very grateful if the esteemed reviewer would reconsider the score.
Thank you for the revision and experiments. The closed loop lowers both harmful bypass and benign acceptance compared with fixed schedules. And joint training shows the same pattern compared with sequential training. I will raise my score from 1 to 2.
We thank the reviewer for their time and effort. We ran the comparison the reviewer asked for in W1 and Q1: the closed loop against fixed schedules on the same guard and candidate pool, with compute accounted at every step. We also compared sequential adaptation with joint training on the same data (Q3), explain the chain of Table 3 (Q2), and revised the presentation (W2).
W1 / Q1. Feedback-based selection against a fixed order.
We compared the closed loop with fixed schedules on the same guard (Llama-Guard-3-1B), the same pool of seven Discovery candidates, and the same training recipe. Compute is the training FLOPs of all steps plus the FLOPs of every gate run, which are charged to the loop, in PFLOP (1015 FLOPs). M1 and M2 are measured as in Section 4.5, on the held-out prompts of each attack. The schedules train on different subsets of the pool, so we average them over the same attacks for every schedule: M1 pool and M2 pool are means over the seven Discovery attacks D1-D7, the pool the gate chooses from, whether or not the schedule trained on them; M1 untrained and M2 untrained are means over the six curated attacks (ReNeLLM, BitBypass, LogiBreak, RespAttack, FITD, Humanize), which no schedule trains on.
"Stopped at the loop's budget" is the fixed schedule after its last step that fits the mean compute of the loop. M1: harmful bypass, lower is better; M2: acceptance of wrapped benign prompts, higher is better
The closed loop reaches the lowest M1 of all schedules in the table. On the untrained attacks both loop rows are below every fixed schedule: 11.6% and 12.9% against 15.1% or higher. The same holds on the pool. With the ranked order the loop is ahead of the full fixed schedule in each of the three seeds, on both sets. At matched compute, a fixed order is far behind, and it does not catch up even when it trains on all seven attacks. At the compute of the loop it is at 17.0% on the pool and 15.1% on the untrained attacks (21.9% and 22.4% with random orders). Trained to the end, at 2.2 times the compute of the loop, it reaches 9.6% and 18.8%, which is still 2.5 and 7.2 points above the loop.
The loop approach is better because after each step the gate is re-run against the adapted guard, and the next candidate that still passes it is trained on; the loop stops when none passes. It trains on two to four of the seven candidates (D2 and D3 in every run) and spends no steps on attacks the guard already blocks. Applying the gate once to the base guard is not enough: it gives 10.3% and 17.8%.
The ranked order also helps: stopped at the loop's budget, a fixed schedule is well ahead with the ranked order (15.1% against 22.4% on the untrained attacks), because the ranking puts stronger candidates first. Once all seven attacks are trained on, the order no longer matters.
Moreover, the loop does not lower M2, it leaves the acceptance of wrapped benign prompts at the level of the base guard or above, on the pool and on the untrained attacks. The fixed schedule over all seven attacks raises it to 91.2% on the pool, because it also trains on the wrappers of the attacks the guard already blocks.
Q2. Composition and order of the chain in Table 3.
Both are arbitrary. Table 3 does not test selection, it tests whether sequential adaptation holds over many steps. The chain consists of seven of the thirteen attacks in an arbitrary order, and the other six are held out to measure transfer. Selection by the gate is tested separately, on Llama-Guard (Table 4) and in the comparison above (W1 / Q1).
The order has little effect on the outcome: in the comparison above, random orders of the same seven attacks end at the same M1 as the ranked order once all seven are trained on (8.9% against 9.6% on the pool). In Table 3 itself, every attack of the chain ends at or below 4.1%, whichever step it was trained at.