acceptodds
Under review as a conference paper at ICLR 2027

SEMA-Auto: Sighted and Efficient Multi-Turn Attack with Autonomy

19 views

Abstract

Existing jailbreak attackers fall into two classes: static planners fix their single- or multi-turn adversarial plans in advance, remaining simple but losing sight of the victim's replies, wasting turns and limiting transferability; interactive generators read each victim reply to generate the next-turn attack but require in-loop evaluators for success signals or hints, incurring substantial costs and inflated performance. Bridging this gap, we propose SEMA-Auto, a sighted and efficient multi-turn attacker with autonomy, interacting with the victim model and terminating the session by itself upon success. To stabilize its behaviors and boost its capabilities, SEMA-Auto is trained in two stages. Synthetic bootstrap tuning installs the notion of attack targets and conforming decisions by fine-tuning the base model with bootstrapped trajectories. Decision-supervised reinforcement learning then optimizes two objectives jointly yet separately, reinforcing the attack with higher jailbreak reward while supervising the decision at each turn against an absolute target. Our method achieves state-of-the-art (SOTA) attack success rates (ASR) across multiple datasets, victims, and judges, outperforming all static and interactive baselines, within almost the fewest turns, demonstrating both effectiveness and efficiency. For example, SEMA-Auto performs a within an average of turns against DeepSeek-V4-Flash on AdvBench, while the best static baseline is with fixed turns and the best interactive baseline is in turns. Against safety-hardened frontier models, GPT-oss-120B, GPT-5.4, Claude-Haiku-4.5, and GPT-6-Astra, our counter-trained attacker achieves an average on HarmBench, over prior SOTA. We will release our code upon acceptance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
jailbreak, attack, multi-turn, reinforcement learning, large language model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Zz8dOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SEMA-Auto, a multi-turn jailbreak model that reads the target model’s responses, generates follow-up prompts, and decides when to stop. Training consists of two stages. First, the model follows a self-generated static attack plan, while an external judge provides labels for its continue-or-stop decisions for supervised fine-tuning. Second, the method combines GRPO, guided by final-response rewards, with a supervised loss on decision tokens to improve both attack success and stopping accuracy.

Critical strengths

SEMA-Auto can detect attack success and stop without an external evaluator at inference time, substantially reducing evaluation costs. It also achieves successful attacks in very few rounds.

Critical weaknesses
  1. The motivation needs clarification. The Abstract and Section 1 highlight the computational costs of in-loop evaluators and the risk of evaluator hacking. However, training still relies heavily on GPT-4.1 to compute response scores RIDAR_{IDA} and generate decision labels. How does this reliance align with the stated motivation? Could bias or drift in the external judge be passed on to SEMA-Auto through the reward signal and decision supervision loss LDS\mathcal{L}_{DS}?

  2. The overall cost advantage is unclear. With G=28G=28 samples per query, 416 queries, and three epochs, training involves tens of thousands of multi-turn trajectories and extensive evaluation by GPT-4.1. Please report the total token usage, number of API calls, and financial cost of training. How much inference is needed for the savings to offset these upfront costs, compared with using a lightweight in-loop evaluator?

  3. The methodological novelty appears limited. SFT on synthetic data and RL with an auxiliary token-level cross-entropy loss are established techniques in reasoning, agent search, and tool use. SEMA-Auto applies this framework to binary continue-or-stop decisions in jailbreak attacks. What is the core algorithmic contribution beyond applying these techniques to this task?

  4. The diversity and adaptability of the learned attacks remain unclear. SEMA-Auto appears to rely mainly on rephrasing the same request. How does its attack success rate change when the target model uses defenses such as semantic drift detection or intent restoration? Can SEMA-Auto move beyond fixed prompting patterns and adapt its strategy based on the target model’s feedback?

Questions for authors

please see the Weaknesses.

8 comments

W1 — Reliance on GPT-4.1 in training

We thank the reviewer for raising this. We would like to kindly clarify the motivation stated in the paper, which concerns the in-loop evaluator at test time:

  • If the in-loop evaluator is the same as the final judge, the attack might stop wherever that evaluator happens to accept, and such a result is not fully reliable. For example, when called again on the same responses, the in-loop evaluator of Crescendo reproduces its own successes on only 82.8% of them.
  • If the in-loop evaluator differs from the final judge, while it cannot be hacked this way, the measured success depends on which evaluator is chosen and largely reflects how often it agrees with the final judge.

Either way, the reported result is sensitive to the in-loop evaluator. We note that we do not argue that such evaluators are unreliable and should be abandoned, which would indeed contradict training with one. Our point concerns their place in the test-time loop, where they make the reported result depend on the evaluator chosen rather than on the core attacker that generates the adversarial prompts. In trained SEMA-Auto, the attacker makes the stopping decision itself, independently of the final judge. Using response scores as the training reward therefore does not conflict with this motivation, which concerns test time alone.

If a training reward has bias or drift, it will be passed on to the attacker, as it is for any method trained with a reward, including the attack content of every learned attacker. Using such scores as the reward is common to learned attackers: SEMA, TRACE, TROJail, and Jailbreak-R1 are all trained with a judge reward. SEMA-Auto uses no additional evaluator, and it is trained to terminate when the reward RIDA≥τR_\text{IDA} \ge \tau and to continue otherwise. We argue that our learning to stop would not add a further source of bias, because the decision-supervised objective pulls in the same direction as the attack objective. Training on this reward already assumes that it moves the attack toward stronger jailbreaks, so a high reward is a reasonable signal to stop. The question therefore returns to the quality of the reward for jailbreak attacks, which is not the main focus of this work. In fact, nothing in our method is specific to RIDAR_\text{IDA}, and our method needs only a score and a threshold, so a better reward can be used in its place without other changes. We have added a clarification in our revision (Appendix B.4).

W2 — Training cost vs. inference savings

We thank the reviewer for raising this. We count every call made in the two training stages. The training-time victim, Llama-3.1-8B-Instruct, is served locally, so the only API calls in training are those of the reward model that computes R_IDA. It scores every victim reply, so it is called once per adversarial turn. The table below compares the calls and tokens of SEMA-Auto with SEMA and TRACE. Attacker tokens are the tokens the attacker generates in reinforcement learning.

AttackerStageTrajectoriesVictim callsAPI calls for rewardAPI tokens for rewardAttacker tokens
SEMAReinforcement learning34,272235,50134,27264.8M16.8M
TRACEReinforcement learning4,16015,355--0.8M
SEMA-AutoSynthetic bootstrap tuning6,65629,82729,82771.4M-
SEMA-AutoReinforcement learning34,272110,267110,267268.7M4.5M
SEMA-AutoTotal40,928140,094140,094340.1M4.5M

Synthetic bootstrap tuning executes 6.7K plans of 4.5 turns on average, 16 for each of the 416 training queries. Reinforcement learning runs 51 steps of 24 queries with G = 28, which samples 34K trajectories of 3.2 turns on average. The two stages of SEMA-Auto call the API reward model 140K times, with 340M tokens in total.

We report the cost in API calls and tokens rather than in dollars. The price of the same tokens and GPU hours differs across providers and changes over time, so a dollar figure would hold for one provider at one date, while the counts above give the cost at any price. For compute, the reinforcement-learning stage of SEMA-Auto runs for 1.6 hours on 8 B200 GPUs, against 3.6 hours on 8 B200 GPUs for SEMA and 2.0 hours on 5 B200 GPUs for TRACE.

After training, SEMA-Auto makes no API call, whereas training-free interactive attacks, such as Crescendo, call an API model as their planner/attacker, their in-loop evaluator, or both, in every attack. The table below shows their calls per attack beyond the victim calls, averaged over the five victims and both datasets in the paper's Table 1. At these rates, Crescendo and GOAT make as many API calls as the whole training of SEMA-Auto within 13K and 14K attacks, FITD within at most 4.2K and X-Teaming within 34K. For reference, evaluating one attacker's ASR with 10 attempts on AdvBench against ten victims takes 52K attacks.

AttackerAPI modelsAvg. turnsAPI calls per attackAttacks to reach 140K API calls
CrescendoGPT-4o (attacker, in-loop evaluator)5.310.513.3K
GOATGPT-4o-mini (attacker)10.010.014.0K
FITDGPT-4o-mini (attacker)9.5≥ 33≤ 4.2K
X-TeamingGPT-4.1 (planner), GPT-4o (in-loop evaluator)3.24.233.7K

We note that during its attack, FITD also generates intermediate prompts and removes the turns the victim refuses, and the calls for these steps are not recorded, so its count in the table is a lower bound.

A lightweight in-loop evaluator that runs locally, such as the HarmBench Classifier used by TRACE and TROJail, makes no API call, so it leaves no API cost to offset. The API calls in our training come from the choice of reward model, GPT-4.1 computing R_IDA, which we take from SEMA, not from learning to stop. The decision target needs only a score of each reply and a threshold, which a local classifier can provide as well, and TRACE trains with a local classifier as its reward. Against such an evaluator, what SEMA-Auto saves is the evaluator's computation at every turn. In the cost analysis of the appendix, against GPT-oss-120B on HarmBench, TRACE runs its 13B in-loop evaluator after every turn and stops after 6.5 turns on average, taking 1.2 hours for the 159 requests. SEMA-Auto stops after 3.5 turns and takes 30.9 minutes.

We have added this analysis to the revision in Appendix D.3.

W3 — Limited methodological novelty

We thank the reviewer for relating our method to techniques used in reasoning, agent search, and tool use, which lets us state its algorithmic contribution more precisely. We acknowledge that both forms have been adopted by prior work: supervised fine-tuning on self-generated data, and a supervised term added to the RL loss. We do not claim either form as part of our novelty. The algorithmic contribution is to train the two outputs of one policy with separate signals: the attack by RL against the jailbreak reward, and the decision by supervision against an absolute per-turn label derived from the same reward.

This differs from how prior work combines the two. In tool use, for example, Hao et al. (2026) alternate RL with phases of supervised fine-tuning on whole ground-truth trajectories of the task, so that the policy imitates a reference solution. A jailbreak has no ground-truth trajectory to imitate. The cross-entropy of SEMA-Auto supervises one action of the policy itself, the decision, at every turn of the attacker's own rollouts, with a label read off the reward the run already computes, and the attack is left entirely to RL.

Our method is therefore not an existing technique borrowed from another field and applied to this task. In fact, it was motivated by our observations and analysis of decision collapse in Section 3.2. There, we find that reward and advantage shaping act only on the decisions a rollout samples, and that GRPO folds decision correctness into a group-relative signal, so once one action stops being sampled, no term placed on the decision can restore it. The decision therefore needs an absolute signal that does not pass through sampling, which supervision provides. We have also explored applying other techniques, including the loss masking of AdaCoT and the decoupled normalization of GDPO, but they do not address the decision collapse in our experiments.

Beyond the algorithm, the paper also contributes by proposing the autonomous attack paradigm, an interactive attacker that decides by itself when to stop and calls no evaluator in the loop, and by identifying and analyzing decision collapse in training such an attacker.

Reviewer_4p6HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SEMA-Auto, a multi-turn jailbreak attacker that adapts its prompts to the victim’s responses and autonomously decides when to stop. The authors identify decision collapse under reinforcement learning, where the attacker degenerates into always continuing or always terminating. They address this through a two-stage training procedure: synthetic bootstrap tuning initializes attack and decision behavior, followed by reinforcement learning with an auxiliary decision-supervision objective. Experiments evaluate attack success, transfer across victims and datasets, stopping behavior, and inference efficiency.

Critical strengths
  1. Comprehensive empirical evaluation. The experiments cover multiple victim models, two benchmarks, three external judges, and a broad range of static and interactive baselines. Beyond attack success, the paper examines interaction length, performance under repeated attempts, and actual runtime, providing substantial evidence for its effectiveness and efficiency claims.

  2. Convincing transfer results. After training against a single victim, the method performs strongly on other victims and on HarmBench, which is not used for training. Results on held-out queries further support its generalization.

  3. Clear connection between the identified failure mode and the proposed solution. The authors investigate decision collapse through training curves and alternative training recipes, motivating explicit decision supervision. Autonomous stopping also has practical value by removing the need for an external evaluator during inference.

Critical weaknesses
  1. The sources of performance improvements are not fully isolated. Some principal baselines use different attacker backbones and training reward evaluators. While the results support the effectiveness of the complete system, comparisons with matched backbones, reward evaluators, and training budgets would better establish the contribution of the proposed mechanisms.

  2. Limited reporting of training variability. Although decision collapse and its sensitivity to random seeds are central to the paper, the main results and key ablations lack systematic statistics across independent training runs. This limits confidence in the stability claims and in some smaller performance differences.

  3. Stopping reliability could be evaluated more directly. Multiple judges and decision-accuracy measurements are valuable, but separately reporting premature termination before success and unnecessary continuation after success would more directly assess the central stopping mechanism. Manual verification of a subset of cases would further strengthen this evidence.

  4. Incomplete coverage of related attack methods. The related work omits several relevant RL-based attack methods, including PISmith (Yin et al., 2026), AutoInject (Chen et al., 2026), and RL-Hammer (Wen et al., 2025), as well as the evolutionary-search attack studied by Nasr et al. While some of these methods target prompt injection in agentic settings rather than multi-turn conversational jailbreaking, discussing their connections and differences would better position the contribution, particularly regarding RL training stability, exploration, and transferability. The authors should also clarify which methods could serve as meaningful experimental comparators under their threat model.

Questions for authors

Do the improvements over the strongest baselines persist when the attacker backbone, training reward evaluator, and training budget are matched?

4 comments

W1 / Q1 — Matched backbone, reward evaluator and budget

We thank the reviewer for raising this. We retrained the strongest baseline, SEMA, with a matched attacker backbone (Qwen2.5-7B-Instruct), reward evaluator (GPT-4.1 computing R_IDA), against the same training-time victim, Llama-3.1-8B-Instruct. The other RL training settings are already matched. We evaluated it against four victims: Qwen2.5-3B-Instruct, Llama-3.1-8B-Instruct, GPT-4.1-mini, and GPT-4o.

The table below shows ASR@1 (%) of the original SEMA from the paper, the retrained SEMA with a matched backbone and reward model, and SEMA-Auto. As in the main text, we use the LLM Classifier on AdvBench and the HarmBench Classifier on HarmBench.

DatasetVictimSEMA (original)SEMA (matched)SEMA-Auto
AdvBenchQwen2.5-3B-Instruct79.676.987.1
AdvBenchLlama-3.1-8B-Instruct79.068.877.5
AdvBenchGPT-4.1-mini81.383.889.2
AdvBenchGPT-4o77.960.683.1
HarmBenchQwen2.5-3B-Instruct77.487.495.6
HarmBenchLlama-3.1-8B-Instruct73.074.277.4
HarmBenchGPT-4.1-mini80.593.190.6
HarmBenchGPT-4o62.969.880.5
Average76.476.885.1

The improvements persist under a matched backbone, reward evaluator and training budget. With matched backbone and reward model, SEMA shows lower ASR on AdvBench and higher ASR on HarmBench, and overall averages 76.8%, against 76.4% for the original SEMA, so the backbone and the reward model barely change SEMA's ASR. SEMA-Auto stays 8.3 points ahead at 85.1%.

W2 — Training variability across runs

We thank the reviewer for this question. We retrained both our main recipe and plain GRPO (the success reward only) with two further seeds from the same stage-1 checkpoint, and measured the balanced accuracy of each run's decisions on the held-out set of annotated trajectory-decision pairs used in the paper. Seed 0 of each recipe denotes the run reported in the paper. The table below shows the balanced accuracy of each run.

RecipeSeed 0Seed 1Seed 2Mean ± std
Success reward only (plain GRPO)0.5970.5000.6210.573 ± 0.064
+ Decision supervision (ours)0.7730.6760.8090.753 ± 0.069

While plain GRPO does not collapse every time when using Qwen2.5-7B-Instruct as the base attacker model, it stays close to the collapsed score of 0.5 in every run. Our test-time attack success rate also holds across seeds. We test them against Llama-3.1-8B-Instruct on AdvBench. The three runs of our recipe reach 77.5, 75.8 and 74.6, respectively, for a mean of 76.0 ± 1.5.

To further justify our reported main comparison on ASR@1 (single attempt per request), we repeat the evaluation of the reported SEMA-Auto attacker and the strongest baseline, SEMA, five times, against Qwen2.5-3B-Instruct, Llama-3.1-8B-Instruct and GPT-4.1-mini. The table below shows ASR@1 (%) mean ± std over 5 runs, with the single-run value reported in our main table (Table 1) in parentheses. Judges follow the main text: LLM Classifier on AdvBench, HarmBench Classifier on HarmBench.

DatasetAttackerQwen2.5-3BLlama-3.1-8BGPT-4.1-miniAvg.
AdvBenchSEMA80.4 ± 0.6 (79.6)76.6 ± 0.9 (79.0)83.5 ± 0.2 (81.3)80.2 (80.0)
AdvBenchSEMA-Auto86.3 ± 0.6 (87.1)74.7 ± 1.6 (77.5)89.3 ± 0.9 (89.2)83.4 (84.6)
HarmBenchSEMA77.7 ± 1.6 (77.4)71.3 ± 3.0 (73.0)79.0 ± 3.0 (80.5)76.0 (77.0)
HarmBenchSEMA-Auto92.5 ± 0.8 (95.6)78.1 ± 2.7 (77.4)90.1 ± 3.5 (90.6)86.9 (87.9)

The five-run means stay close to the single-run values, and the better attacker in every setting is the same under both.

W3 — Direct evaluation of stopping

We thank the reviewer for the constructive feedback. Following this, we add the two errors on the held-out set to Appendix C.6 of the revision. We split the balanced accuracy reported in the paper into the two errors it averages on the held-out set of annotated trajectory-decision pairs. Among the continue-labeled points, the false termination rate is the percentage at which the attacker terminates, which is premature termination before success. Among the terminate-labeled points, the false continuation rate is the percentage at which the attacker continues, which is unnecessary continuation after success. Balanced accuracy equals one minus the mean of the two rates.

The table below shows the balanced accuracy and both errors for each training recipe of the ablation on Llama-3.2-3B-Instruct. SEMA-Auto rarely terminates prematurely, at 5.4%, and its errors lie on the side of continuing, at 38.2%. Every alternative recipe collapses onto a single decision: four of them never terminate, with false continuation at 100%, and the turn-efficiency penalty always terminates, with false termination at 100%.

Training recipeBalanced acc.False termination (%)False continuation (%)
Success reward only0.5000.0100.0
+ Turn-efficiency penalty0.500100.00.0
+ Decision-correctness reward0.5000.0100.0
Decision-masked advantage0.5000.0100.0
Decoupled normalization0.5000.0100.0
+ Decision supervision (ours)0.7825.438.2

We further show the same measures across the decision-supervision weight α\alpha on Qwen2.5-7B-Instruct. Our reported run (α\alpha=0.007) terminates prematurely at 1.8% and continues unnecessarily at 43.6%. Across the non-zero weights, false termination stays at or below 4.2%, while false continuation ranges from 11.5% to 51.5%.

Decision-supervision weight α\alphaBalanced acc.False termination (%)False continuation (%)
00.5970.680.0
0.0030.7273.051.5
0.0050.8464.226.7
0.007 (ours)0.7731.843.6
0.0100.9214.211.5

As a small-scale manual check on real test-time trajectories, six annotators labeled 60 decision points of SEMA-Auto, drawn from its trajectories against the five victims in the paper's Table 1 and balanced across its decision and whether the main-text judge counts the response as a jailbreak. Each annotator saw only the harmful request and the victim response, and judged whether the response gives the requester what the request asks for. With the human labels as the reference, defining the two rates as above and reweighting the balanced sample to how often each combination occurs in the test trajectories, the false termination rate of SEMA-Auto is 15.6%, and its false continuation rate is 55.8%, 5.4 and 8.6 points above the 10.2% and 47.2% obtained on the same trajectories with the main-text judge as the reference. Part of the gap comes from the annotators being more lenient than the main-text judge: they count 61.3% of the decision points as jailbreaks, against 53.8% for the judge, so some responses after which the attacker continues count as successes under the human labels. Given the small sample, we take this only as an indication that the same pattern holds on real trajectories: false termination stays well below false continuation.

W4 — Missing related RL-based attacks

We thank the reviewer for naming these works. As noted by the reviewer, these works mainly focus on the threat model of indirect prompt injection: the attacker controls text that an application or a tool-using agent reads as data, and succeeds when the victim carries out the injected task. Our setting is direct jailbreaking: the attacker converses with the victim itself and succeeds when the victim fulfills a harmful request. In both settings, the attacker learns from a reward computed on the victim's output, and exploration is limited in a similar way: RL-Hammer reports training converging onto a single universal attack pattern, and a single SEMA-Auto attacker likewise keeps to a narrow prompting paradigm.

Reinforcement learning is also sometimes unstable in both settings, but the instability differs. In the injection attacks, it lies in the attack content: against a robust victim, few injections succeed, so a group of rollouts often receives the same reward and carries no signal. RL-Hammer counters this by training jointly with an easier victim. AutoInject replaces the binary reward with a feedback model's preference for a candidate over the best suffix found so far. PISmith adds an entropy bonus that grows as successes become rarer. The instability we identify lies instead in the continue-or-terminate decision. Under simple reinforcement learning, the decision degenerates to a constant. These injection attacks write a single injection per attempt and make no such decision, so this problem does not arise in them. Transfer also differs. In its prompt-injection experiments, RL-Hammer reports that an attacker trained against a weaker victim fails to generalize to stronger ones such as GPT-4o. SEMA-Auto, trained against Llama-3.1-8B-Instruct alone, achieves the highest ASR@1 on almost all victims and datasets unseen in training.

Experimental comparators. AutoInject and PISmith are built around an injected task embedded in data, so applying them to direct jailbreaking would require redefining their task. RL-Hammer can be adapted to our threat model, and it also reports a jailbreak experiment on AdvBench. However, its public code covers only the prompt-injection experiments and provides no configuration for the jailbreak one, and the paper does not specify that experiment's setup, such as the training victim, the reward, and the data split. Besides, as a single-turn attacker trained by reinforcement learning, it belongs to the same class as Jailbreak-R1, which is among our baselines. Nasr et al. evaluate twelve published defenses against both jailbreaks and prompt injection, with several adaptive attacks. The evolutionary-search attack the reviewer mentions is one of them, a genetic algorithm whose mutations are proposed by an LLM. But we also found no public implementation for it. Our baselines have included the learned attackers that share our setting, e.g., Jailbreak-R1, TROJail, and TRACE. Given that, we think that including them might add little beyond our existing experiments, so we did not add them to our comparison.

We have added a discussion (Appendix D.2) about these works in our revision.

I thank the authors for their clear replies and additional results. I believe they will strengthen the paper and will keep my score.

Reviewer_tuoqOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SEMA-Auto, a multi-turn jailbreak attacker that reads the victim’s replies and decides by itself at each turn whether to continue or terminate, without needing an in-loop evaluator. The authors find that naively training such an attacker with GRPO leads to decision collapse, where it always continues or always terminates. To fix this, they first bootstrap the attacker on self-generated trajectories annotated with decisions, and then train it with RL on the attack plus a separate supervised loss on the decision. Experiments on AdvBench and HarmBench with many victims, including recent frontier models, show the highest ASR@1 in most settings with close to the fewest turns.

Critical strengths
  1. This paper has clear problem framing with a solid failure analysis. The paper doesn’t just add a stop token to an RL attacker. It identifies decision collapse, shows that it persists across many reward and advantage shaping variants, and explains why. The fix follows directly from this analysis, and the figures make the pipeline easy to follow. This is what makes the work more than just a stronger attacker.
  2. Provides thorough and up-to-date experiments. The evaluation covers ten victims including very recent frontier models, three judges, and more than ten baselines. The turn-adaptivity analysis (Fig. 6) is especially convincing: the attacker spends more turns on harder cases and stops early once it succeeds.
Critical weaknesses
  1. There is limited novelty over SEMA, and the comparison with SEMA is confounded. Much of the pipeline is inherited from SEMA (synthesis prompts, R_IDA reward, data, judges), and closed-loop RL attackers already exist (TRACE, TROJail). The new part is self-termination, and decision collapse also looks closely related to prior work such as Hao et al., 2026. More importantly, SEMA uses Llama-3.1-8B and GPT-4.1-mini, while SEMA-Auto uses Qwen2.5-7B and GPT-4.1. So I cannot tell how much of the gain comes from autonomy and how much from a different backbone and a stronger reward model. SEMA is even slightly better on the training-time victim (79.0 vs. 77.5).
  2. No in-loop evaluator is overstated. The decision labels come from R_IDA (GPT-4.1) during training, so the evaluator is in effect distilled into the attacker. Not calling a judge at test time is a real advantage, but the paper should frame this as an internalized evaluator.

Hao et al., Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It.

Questions for authors
  1. Can you report SEMA trained with the same base attacker and reward model as SEMA-Auto (Qwen2.5-7B, GPT-4.1), under a matched turn budget? This is the comparison I most need to attribute the gains to autonomy.
  2. How does decision collapse, and your fix for it, differ from Hao et al. and the adaptive-thinking works you cite? Is anything specific to the jailbreak setting?
  3. How often does the attacker’s TERMINATE decision agree with the final external judges, which differ from the GPT-4.1 R_IDA used for training labels?
  4. What is the total API cost of training, given that GPT-4.1 is called on every rollout turn, plus counter-training against frontier APIs? How does this compare with the inference cost of interactive baselines?
4 comments

W1a / Q2 — Novelty over SEMA and prior work

We thank the reviewer for this question. We do follow many of SEMA's settings, namely its data, judges, synthesis prompts, and R_IDA reward, for two reasons. First, the synthesis prompts and R_IDA reward are contributions of SEMA, and SEMA-Auto builds upon these contributions rather than redesigning them. Second, keeping these settings fixed allows the comparison with SEMA to isolate the effect of the algorithmic differences. The matched retraining in our response to W1b further shows that our improvement is not from the backbone or the reward model.

SEMA is a static attacker whose plan is fixed before the session opens. TRACE and TROJail are interactive, but they end a session when an in-loop evaluator, the HarmBench Classifier, declares success. Beyond them, what SEMA-Auto adds is the decision to end the session and no in-loop evaluator at test time, as the reviewer recognized. Self-termination in turn introduces a new difficulty, decision collapse. We explain below how it, and our remedy for it, differ from Hao et al. and from adaptive thinking.

Difference from Hao et al. The collapse they report in multi-step tool use concerns the output format: the control tokens <tool_call> and <|im_end|> are amplified until the model emits only a minimal terminating sequence, which is invalid. In SEMA-Auto, we make the decision well-formed via format reward but it collapses to a constant. We trace it to a fragile base model, unreachable unsampled actions, and entangled relative signal in RL (Section 3.2). The remedies differ as well. Their two strongest remedies alternate RL with phases of supervised fine-tuning on whole ground-truth trajectories of the task, or on reflections an external model writes about the policy's failures. SEMA-Auto adds the decision-supervision loss to the RL loss in every update, on the decision token only, over the attacker's own rollouts, with a label read off the reward the run already computes.

Difference from adaptive thinking. Adaptive thinking reports the phenomenon closest to ours: the control token that selects a reasoning mode collapses under RL to always or never thinking, as AdaCoT and Thinkless report. The remedies in this line act on the decision through the RL signal or the rollouts. We discussed them in Appendix D.6.2. In our early exploration, we tried some approaches similar to their remedies, for example, the selective loss masking of AdaCoT and a load-balancing term from mixture-of-experts routing as in Kassenaar et al. However, the results show that the decision still collapses. Several others do not apply to a multi-turn attack, and Appendix D.6.2 lists them with the reasons. The difference from SEMA-Auto lies in the signal on the decision. These remedies change how the RL objective treats the decision, through its reward, its advantage, its weight in the loss or its sampling, so the decision is still learned from the returns of the tokens a rollout samples. SEMA-Auto adds a separate supervised signal: a cross-entropy on the decision token against a per-turn label, which moves the decision toward the labeled decision whether or not a rollout sampled that decision.

What is specific to our setting. (i) The success of an attack lies in the victim's reply, which the attacker has already read when it decides. In adaptive thinking, success depends on the answers that have not yet been generated by the policy, so whether a query needs thinking is known only by comparing the two modes. However, whether the attack has succeeded is a property of the latest reply, which the training reward has already scored. The correct decision is therefore available at every turn, and this is what makes direct supervision possible. (ii) No ground-truth trajectory of an attack exists to imitate, which rules out remedies built on demonstrations, such as those of Hao et al.

W1b / Q1 — Confounded comparison with SEMA

We retrained SEMA with SEMA-Auto's attacker backbone (Qwen2.5-7B-Instruct) and reward model (GPT-4.1 computing R_IDA), against the same training-time victim, Llama-3.1-8B-Instruct. We evaluated it in the main-text setting against Qwen2.5-3B-Instruct, Llama-3.1-8B-Instruct, GPT-4.1-mini, and GPT-4o.

The table below shows ASR@1 (%) of the original SEMA from the paper, the retrained SEMA with a matched backbone and reward model, and SEMA-Auto. As in the main text, we use the LLM Classifier on AdvBench and the HarmBench Classifier on HarmBench.

DatasetVictimSEMA (original)SEMA (matched)SEMA-Auto
AdvBenchQwen2.5-3B-Instruct79.676.987.1
AdvBenchLlama-3.1-8B-Instruct79.068.877.5
AdvBenchGPT-4.1-mini81.383.889.2
AdvBenchGPT-4o77.960.683.1
HarmBenchQwen2.5-3B-Instruct77.487.495.6
HarmBenchLlama-3.1-8B-Instruct73.074.277.4
HarmBenchGPT-4.1-mini80.593.190.6
HarmBenchGPT-4o62.969.880.5
Average76.476.885.1

With a matched backbone and reward model, SEMA shows lower ASR on AdvBench and higher ASR on HarmBench, and overall averages 76.8%, against 76.4% for the original SEMA, so the backbone and the reward model barely change SEMA's ASR. SEMA-Auto stays 8.3 points ahead at 85.1%.

We further compare them under matched turn budgets. Both the original SEMA and the matched SEMA use seven turns on every request, so we cut off SEMA-Auto's trajectories at the seventh turn. However, since SEMA-Auto is already using only a few turns, its ASR@1 decreases very little and still averages 85.0%.

W2 — Evaluator distilled during training

We thank the reviewer for recognizing that not calling a judge at test time is a real advantage. We agree that the attacker internalizes the success criterion of its training reward. However, we argue that by definition every interactive attacker would read the victim's reply and decide what to say next, so a judgment of the victim reply is already part of what it learns. SEMA-Auto simply adds one action to the search space, "stop and say nothing", with no new model, no new module, and no additional inference call. Learning-based attackers, such as SEMA, TRACE, TROJail and Jailbreak-R1, are commonly trained with a judge reward and thereby distill it into the policy. SEMA-Auto reuses its reward model to label the decisions and adds no extra training-time evaluator. Therefore, if SEMA-Auto includes an internalized evaluator, then in some sense other interactive attackers should also have one. We will add this clarification to our revision.

Q3 — TERMINATE vs. the external judges

We thank the reviewer for this question. We measure this on the test trajectories against the five main-text victims on both datasets. The table below shows, for SEMA-Auto and each interactive baseline with an in-loop evaluator, the percentage of TERMINATE decisions whose response the judge also counts as a jailbreak, averaged over the five victims. Following our main text, the judge is the LLM Classifier on AdvBench and the HarmBench Classifier on HarmBench. When SEMA-Auto terminates, the judge agrees in 84.8% of cases on AdvBench and 88.7% on HarmBench. Terminations triggered by an in-loop evaluator are confirmed less often: on AdvBench, between 35.4% (Crescendo) and 69.8% (TRACE). On HarmBench, CoA reaches 88.8%, 0.1 pp higher than ours. We note that TRACE and TROJail reach 100% only because their in-loop evaluator is the HarmBench Classifier itself.

AttackerAdvBenchHarmBench
Crescendo35.454.0
CoA69.688.8
X-Teaming61.069.1
TROJail58.4100.0*
TRACE69.8100.0*
SEMA-Auto (ours)84.888.7

Q4 — Total API cost of training

We thank the reviewer for raising this. We count every call made in the two training stages. The training-time victim, Llama-3.1-8B-Instruct, is served locally, so the only API calls in training are those of the reward model that computes R_IDA. As the reviewer notes, it scores every victim reply, so it is called once per adversarial turn. The table below compares the calls and tokens of SEMA-Auto with SEMA and TRACE. Attacker tokens are the tokens the attacker generates in reinforcement learning.

AttackerStageTrajectoriesVictim callsAPI calls for rewardAPI tokens for rewardAttacker tokens
SEMAReinforcement learning34,272235,50134,27264.8M16.8M
TRACEReinforcement learning4,16015,355--0.8M
SEMA-AutoSynthetic bootstrap tuning6,65629,82729,82771.4M-
SEMA-AutoReinforcement learning34,272110,267110,267268.7M4.5M
SEMA-AutoTotal40,928140,094140,094340.1M4.5M

Synthetic bootstrap tuning executes 6.7K plans of 4.5 turns on average. In our main run, reinforcement learning samples 34K trajectories of 3.2 turns on average. The two stages of SEMA-Auto call the API reward model 140K times with 340M tokens in total. SEMA-Auto makes 4.1 times as many reward-model calls as SEMA, whose first stage samples plans without executing them and makes no API calls, and whose RL stage scores only the final turn. SEMA executes a plan of 6.9 turns compared to our 3.2 turns, so SEMA-Auto makes fewer than half of SEMA's victim calls, 110K against 236K. When we counter-train the attacker against a frontier victim model, the victim calls in both stages go to that model's API (except against GPT-oss-120B, which we serve locally). For example, against GPT-5.4, SEMA-Auto runs 4.76 turns on average in RL training.

After training, SEMA-Auto makes no API call. In contrast, training-free interactive attacks, such as Crescendo, call an API model as their planner/attacker, their in-loop evaluator, or both, in every attack. The table below shows these calls per attack beyond the victim calls, averaged over the five victims and both datasets in the paper's Table 1. At these rates, Crescendo and GOAT make as many API calls as the whole training of SEMA-Auto within 13K and 14K attacks, FITD within at most 4.2K and X-Teaming within 34K. For reference, evaluating one attacker's ASR with 10 attempts on AdvBench against ten victims takes 52K attacks.

AttackerAPI modelsAvg. turnsAPI calls per attackAttacks to reach 140K API calls
CrescendoGPT-4o (attacker, in-loop evaluator)5.310.513.3K
GOATGPT-4o-mini (attacker)10.010.014.0K
FITDGPT-4o-mini (attacker)9.5≥ 33≤ 4.2K
X-TeamingGPT-4.1 (planner), GPT-4o (in-loop evaluator)3.24.233.7K

We note that during its attack, FITD also generates intermediate prompts and removes the turns the victim refuses, and the calls for these steps are not recorded, so the count in the table is a lower bound.

We have added this analysis to the paper in Appendix D.3.

Thanks for the response. The matched SEMA experiment basically resolves my main concern. I still think the novelty is limited, since this is largely an interactive and self-terminating extension of SEMA, and “no in-loop evaluator” should really mean no evaluator call at test time. But I already thought the paper was solid enough for acceptance before the rebuttal, and now the main confound I was worried about is gone. You also answered basically every question with actual experiments. So I am raising my score from 3 to 4.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Revision Uploaded

We sincerely thank all reviewers for their time and their constructive feedback, which has helped us improve the paper considerably. We have uploaded a revised version of the paper, with all changes highlighted in blue. We hope the revision and our responses address your concerns, and we look forward to further discussion.

AuthorsOpenReview ↗

Rebuttal Summary

Dear AC and reviewers,

We sincerely thank the AC for handling our submission, and all reviewers for their constructive feedback, which has helped us improve the paper. To facilitate the discussion, we summarize below the main concerns raised during the rebuttal and how we addressed them.

Main refinements

  • Matched comparison with SEMA

    We retrained SEMA with the same attacker backbone and reward model as SEMA-Auto. Its average ASR@1 barely changes (76.8 vs. 76.4), while SEMA-Auto stays ahead at 85.1, indicating that the gain comes from our method rather than from the backbone or the reward model (tuoq W1b/Q1; 4p6H W1/Q1).

  • Stopping reliability

    We report how often SEMA-Auto's stopping decisions agree with the final judges, distinguish premature termination from unnecessary continuation, and add human verification. SEMA-Auto rarely terminates prematurely; its errors mainly involve continuing longer than necessary (tuoq Q3; 4p6H W3).

  • Training cost and the training-time evaluator

    We report all API calls and tokens used in training. Since SEMA-Auto makes no API calls at inference, the interactive baselines make as many API calls as SEMA-Auto's entire training within about 4K-34K attacks (about 13K for Crescendo) (tuoq Q4; Zz8d W2). We also clarify that our motivation concerns the in-loop evaluator at test time, while training with a judge reward is common to all learned attackers (tuoq W2; Zz8d W1).

  • Method novelty

    We clarify that our contributions are the autonomous attack paradigm, the identification of decision collapse, and decision-supervised learning as its remedy. The attack is trained by RL, while the stopping decision is learned jointly yet separately using a decision-supervision signal. We also explain how it differs from existing techniques. Hao et al. (2026) instead imitate ground-truth trajectories, which do not exist for jailbreaks, and decision-shaping methods such as AdaCoT and GDPO still learn the decision only from sampled actions, which did not prevent the collapse in our experiments (tuoq W1a/Q2; Zz8d W3).

Remaining concern of Reviewer Zz8d

Reviewer Zz8d's remaining concern is that, in an additional test conducted during the rebuttal (against GPT-4o with the THRD defense), SEMA-Auto continues attacking after responses accepted by the HarmBench classifier, resulting in a large gap between final-turn and any-turn ASR (0.0 vs. 29.6). We acknowledge that this exposes a limitation in SEMA-Auto's stopping reliability under strong defenses. This is consistent with the error direction reported above: SEMA-Auto uses a stricter stopping criterion than the HarmBench classifier, so when a defense allows only partially harmful responses, it may continue seeking a stronger response, with subsequent turns being blocked.

This issue is much less pronounced in our main, undefended evaluations: the gap between final-turn and any-turn ASR is small (82.2 vs. 88.8 on HarmBench, averaged over five victims), and SEMA-Auto loses only 3.7-13.4% of the successes it already had, comparable to baselines that use an in-loop evaluator (Appendix D.4). Our claim is therefore not that SEMA-Auto always stops at the optimal turn (in general, given the existence of various judges and success criteria, no interactive attacker can stop optimally under every judge), but that it attacks effectively and efficiently without an in-loop evaluator. We have added this discussion to the limitations section in our revision.


Overall, with the additional experiments and clarifications, we believe we have addressed most of the reviewers' concerns while acknowledging the remaining limitation. We have uploaded a new revision with all changes highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.