acceptodds
Under review as a conference paper at ICLR 2027

Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking

Abstract

Deploying LLMs in multi-turn dialogues facilitates jailbreak attacks that distribute harmful intent across seemingly benign turns. Recent training-based multi-turn jailbreak methods learn long-horizon attack strategies from interaction feedback, but often rely on coarse trajectory-level outcome signals that broadcast uniformly to every turn. However, we find that turn-level contributions in multi-turn jailbreaking are non-uniform (only a few turns drive success), phase-dependent (depending on the phase of context) and target-specific (depending on the target model). Such coarse outcome supervision induces a credit assignment problem, leading to over-rewarding redundant turns in successful trajectories and under-crediting useful intermediate turns in failed ones. To address this, we propose **TRACE**, a turn-aware credit assignment framework for reinforcement learning (RL)-based multi-turn jailbreaking. For successful trajectories, TRACE estimates turn-level contributions via leave-one-turn-out semantic masking; for failed ones, TRACE assigns penalties based on prompt harmfulness and semantic relevance, with an additional local refusal-aware penalty. Furthermore, we reuse the attack-side credit signal for multi-turn defense alignment. Extensive experiments on open-source and closed-source targets show that TRACE achieves the strongest overall performance in effectiveness, transferability, and efficiency, yielding an approximately 25% relative improvement in attack success rate over the strongest RL baseline while also improving the safety–utility balance when reused for defense alignment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, multi-turn jailbreaking, reinforcement learning, credit assignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8uE7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the credit assignment problem in RL-based multi-turn jailbreaking and proposes TRACE, a turn-aware framework that redistributes trajectory-level outcome rewards across dialogue turns. The authors empirically identify three properties of turn-level contributions—non-uniformity, phase dependency, and target specificity—and design asymmetric credit mechanisms for successful and failed trajectories. Successful trajectories use leave-one-turn-out semantic masking, while failed trajectories use phase-aware harmfulness and semantic relevance deviations, together with a refusal-aware local penalty. Extensive experiments show substantial improvements over prior multi-turn jailbreak baselines in attack effectiveness, transferability, and inference efficiency. The extracted credit signal is additionally reused for defense alignment.

Critical strengths
  1. The paper identifies a clear and consequential credit-assignment failure mode in existing multi-turn RL training.
  2. The experimental validation is broad and the gains are substantial.
  3. The paper goes beyond attack performance and shows that the same turn-level signal can be useful for defense
Critical weaknesses
  1. The turn-level credit estimator is intervention-dependent rather than a causal decomposition of trajectory return.
  2. The paper does not fully characterize training variance.
  3. The failure-side component is more heuristic than the success-side one.
Questions for authors
  1. For failed trajectories, how much of the improvement comes from the phase-specific prior itself? A comparison with a simpler phase-agnostic harmfulness/relevance reward would help clarify this.
  2. For the success-side credit, what happens if you regenerate the later dialogue after removing a turn, instead of keeping the remaining suffix fixed? Do you get similar turn rankings?
  3. Most experiments use a 5-turn budget. Do you have any results on longer conversations, and how does the extra leave-one-turn-out computation scale with the number of turns?
7 comments

We thank the reviewer for the thoughtful comments and constructive questions. These have motivated additional experiments that provide a more complete evaluation of TRACE and strengthen the empirical support for our conclusions.

W1: The turn-level credit estimator is intervention-dependent rather than a causal decomposition of trajectory return.

We agree with the reviewer's assessment. TRACE's success-side credit is an intervention-dependent directional proxy for turn-level credit assignment.

In multi-turn jailbreaking, the judge provides a trajectory-level harmfulness score: it evaluates whether the final target response fulfills the original harmful request, but does not provide ground-truth labels for each turn's contribution to the outcome. Different turns may depend on one another, jointly establish context, and exert delayed effects. Therefore, the final attack outcome alone does not uniquely determine how credit should be distributed across turns. This is one reason why credit assignment in multi-turn jailbreaking is challenging.

Other credit assignment methods define their signals according to their respective objectives. For example, IGPO defines turn-level rewards through changes in the model's probability of generating a known correct answer, providing a computable progress signal for multi-turn search. SCAR first defines a utility function over text subsets and then uses Shapley values to assign token- or span-level credit. These methods provide credit under their respective definitions, but do not directly yield a unique attribution of final jailbreak success to individual turns.

TRACE fixes the final attacker query xTx_T, masks a non-final interaction pair, and regenerates the final target response to compute credit from the harmfulness-score change. This intervention-relative sensitivity guides turn-level training feedback (Appendix B.1). We evaluate its stability through intervention and repeated-resampling experiments (Appendix B.2), and its training utility through ablations (Table 3). We will clarify this interpretation in the main text.

[1]. Wang et al. IGPO ICLR 2026; arXiv:2510.14967, 2025.

[2]. Cao et al. SCAR arXiv:2505.20417, 2025.

W2: The paper does not fully characterize training variance.

Thank you for this suggestion. We have added experiments to characterize training variance. Using Llama-3.1-8B-Instruct as the target, we independently trained GRPO and TRACE (single) with three training seeds: 10000, 20000, and 30000. The results are reported below as ASR₅@1 (%).

MethodTraining seedHBJBBWJBMacro
GRPO1000074.8472.7370.0072.52
GRPO2000079.2581.8278.0079.69
GRPO3000074.8481.8270.5075.72
GRPOMean ± std76.31 ± 2.5478.79 ± 5.2572.83 ± 4.4875.98 ± 3.59
TRACE (single)1000084.9178.1883.0082.03
TRACE (single)2000078.6283.6480.5080.92
TRACE (single)3000082.3976.3678.5079.08
TRACE (single)Mean ± std81.97 ± 3.1779.39 ± 3.7980.67 ± 2.2580.68 ± 1.49

TRACE (single) achieves a higher Macro ASR than GRPO under all three training seeds, with an average improvement of 4.70 percentage points and a lower cross-seed standard deviation (1.49 vs. 3.59). These results indicate that its performance gains are relatively stable across these three independent training runs.

In addition, Table 6 in Appendix B.2 provides further analysis of the sampling variability of the LOTO credit used during training.

Resampling settingSpearman ρ [95% CI]Raw sign agreementMargin-sign agreement
Temperature = 0, K = 30.891 [0.842, 0.929]90.8%100.0% (116/116)
Temperature = 0.7, K = 100.669 [0.578, 0.741]80.5%97.0% (128/132)

Under the temperature = 0 setting used in training, the credit estimates exhibit high ranking consistency under repeated resampling. With greater sampling randomness, the rankings remain positively correlated, and the signs of credits away from zero also remain highly consistent.

W3: The failure-side component is more heuristic than the success-side one.

We agree that the failure-side turn-aware deviation penalty has a stronger heuristic component than the success-side credit. Its design is motivated by the premature exposure and harmfulness drift observed in Section 3.2, together with the target specificity discussed in Section 3.3.

Specifically, we estimate harmfulness and relevance priors from successful trajectories against the corresponding target, and then assign penalties according to how much each turn in a failed trajectory deviates from these priors (Appendix B.3). This component is therefore an empirically motivated penalty design for policy optimization.

Q1: Ablation for Phase-Agnostic Prior: replacing phase-specific priors with pooled priors

Thank you for this suggestion. It helps us distinguish the role of information derived from successful trajectories from that of phase-specific modeling, making our experimental analysis of the failure-side component more complete. We added two ablations:

  • A1: Phase-Agnostic Prior, directly addressing the contribution of phase-specific modeling.
  • A2: TROJail-Inspired Fixed Heuristic, an additional ablation that replaces the success-derived priors with fixed heuristic references.

Ablation configurations

A1: Pooled Phase-Agnostic Prior

In A1, we retain the prior information derived from successful trajectories but remove its phase dependence. All phases within a trajectory therefore use the same feature references. We keep the other conditions unchanged, reuse the same successful calibration trajectories for each target, and pool the features across phases rather than estimating separate priors for each phase.

The original phase-specific priors and the pooled phase-agnostic priors are shown below.

Harmfulness priors

PriorTurnQwen SafeQwen Cont.Qwen UnsafeOSS SafeOSS Cont.OSS UnsafeLlama SafeLlama Cont.Llama Unsafe
Phase-specific (ours)10.780.080.140.870.110.020.560.130.31
Phase-specific (ours)20.510.140.340.650.280.070.400.210.38
Phase-specific (ours)30.250.190.560.560.250.180.410.180.41
Phase-specific (ours)40.150.240.610.470.260.260.230.220.55
Phase-specific (ours)50.060.140.800.200.340.460.130.400.47
Phase-agnostic (A1)All0.350.160.490.550.250.200.350.220.43

Relevance references

PriorTurnQwenOSSLlama
Phase-specific (ours)10.400.400.52
Phase-specific (ours)20.450.320.52
Phase-specific (ours)30.500.320.46
Phase-specific (ours)40.500.290.50
Phase-specific (ours)50.500.320.48
Phase-agnostic (A1)All0.450.330.49

Intuitively, pooling features across phases makes all turns share the same harmfulness and relevance references. This removes differences between the phase-specific reference distributions and their explicit modeling of turn-by-turn progression. The resulting reward applies the same criteria across phases and cannot express how the appropriateness of the same harmfulness level may differ by phase.

A2: TROJail-Inspired Fixed Heuristic

We added this experiment to further assess the need for TRACE's target- and phase-specific success priors. We remove the target-specific priors used for failed trajectories, so the references no longer depend on successful trajectories against the particular target.

Inspired by the fixed heuristic in TROJail §4.2–4.3,

rh2(xt)=tTcos⁡(e(x0),e(yt)),r_{h_2}(x_t)=\frac{t}{T}\cos(e(x_0),e(y_t)),

we assign a fixed linear reference to each phase. For example, the relevance weights for a four-turn trajectory are 0.25, 0.50, 0.75, and 1.00. Harmfulness also uses a fixed increasing reference: 0.2, 0.4, 0.6, 0.8, and 1.0 for five-turn trajectories, and 0.25, 0.50, 0.75, and 1.00 for four-turn trajectories. Attacker-prompt harmfulness labels are mapped to safe = 0, controversial = 0.5, and unsafe = 1.

However, the prior tables above show that successful trajectories do not necessarily require increasingly related queries in later phases. Instead, successful attacks follow different pacing against targets with different refusal boundaries.

Ablation results

MethodQwen / HBQwen / JBQwen / WJBLlama-3.1-8B-IT / HBLlama / JBLlama / WJBGPT-OSS-20B / HBOSS / JBOSS / WJBAverage
GRPO81.5577.5881.3371.4874.5569.6764.1566.0665.0072.37
A1: Pooled phase-agnostic prior85.5387.2786.0074.8472.7370.5069.1861.8258.0073.99
A2: TROJail-inspired fixed linear reference87.4276.3687.0076.1074.5579.5051.5752.7345.5070.08
TRACE (single)87.8495.1589.8379.6684.2484.0082.8076.9784.1784.96
TRACE (mix)90.5787.7290.5084.4889.0988.6783.6486.0683.1787.10

Replacing the phase-specific prior with the pooled phase-agnostic prior reduces the average ASR from 84.96% to 73.99%. One possible explanation is that a shared prior cannot adequately capture the different harmfulness and relevance requirements across phases, weakening the failure-side credit's ability to distinguish useful scaffolding from inappropriate behavior.

A2 achieves an average ASR of 70.08%, below GRPO's 72.37%. Although the success-side credit remains unchanged, replacing the failure-side design still reduces overall performance. This suggests that a fixed heuristic that does not reflect successful behavior against the specific target may struggle to accommodate target-dependent attack pacing. The current results therefore support TRACE's default design over the fixed heuristic tested in this ablation.

Q2: Evaluating LOTO credit using regenerated dialogue continuations and context-neutral infilling.

Thank you for this suggestion. We added an experiment to examine whether the turn rankings obtained by LOTO depend on the subsequent target responses already generated in the original trajectory.

First, we fix the subsequent attacker queries, including the final attacker query xTx_T, and compare the following two settings:

  • Original LOTO: Remove (xt,yt)(x_t,y_t), retain the subsequently generated interaction pairs, and regenerate only the final response yTy_T.
  • New experiment: Remove (xt,yt)(x_t,y_t), fix the subsequent attacker queries, and regenerate all subsequent target responses yt+1,…,yTy_{t+1},\ldots,y_T.

The new experiment therefore allows subsequent target responses to adapt to the modified history. We conduct it only on successful trajectories, consistent with TRACE's use of success-side credit.

We select 20 successful trajectories from each of the early, middle, and late stages of training, yielding 60 trajectories and 174 non-final turns. For each turn under examination, we first regenerate the subsequent target responses without removing any content and retain 10 successful trajectory realizations, yielding 174×10=1,740174 \times 10 = 1,740 successful realizations in total.

We then apply both settings to the same successful realization. All outcomes after turn removal are retained, regardless of whether the attack remains successful. Both settings use the same harmfulness judge to compute credit. We average the 10 credit estimates for each turn and then compare the turn rankings within each trajectory.

EvaluationResult
Spearman ρ between the two turn rankings, averaged over 60 trajectories0.522, 95% CI [0.343, 0.675]
Agreement in attack-critical / redundant classifications80.2%, 95% CI [76.8%, 84.0%]

We follow the definitions in the paper: for a successful trajectory, a turn is classified as attack-critical if removing it causes the attack to fail, and redundant if the attack remains successful.

Among cases classified as attack-critical by LOTO, 76.5% remain attack-critical after regenerating the subsequent target responses. Among cases classified as redundant by LOTO, 81.7% remain successful after regeneration. Thus, allowing subsequent target responses to update preserves similar turn rankings and the distinction between attack-critical and redundant turns.

Context-neutral infilling with target-response regeneration

We additionally conduct a context-neutral infilling experiment.

Following the principles in Appendix B.2, we preserve the original turn's entities and necessary references as much as possible while removing content that advances the attack. For example, we retain names such as BERT and RankBrain but rewrite the query to ask for the titles and publication dates of related papers. In a trajectory concerning explosive manufacturing, we replace an early scaffolding turn about a chemistry course with a question about a mathematics course. Ordinary background questions are rewritten as ordinary questions, without adding refusals or safety advice.

Both conditions regenerate all target responses from turn tt onward:

  • Original-query replay: Retain the original attacker query xtx_t at turn tt.
  • Context-neutral infilling: Replace xtx_t with an ordinary question that does not advance the attack while preserving contextual continuity.

All other attacker queries, including xTx_T, remain fixed. The difference between the two conditions therefore helps assess whether the attack-progressing content in the original turn continues to contribute.

Setting / comparisonAttack success rate / difference
Original-query replay76.26%
Context-neutral infilling64.20%
Reduction relative to the replay control12.07 percentage points, 95% CI [7.63, 17.04]
Reviewer_4stFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TRACE. TRACE improves multi-turn jailbreak training by identifying which dialogue turns drive success, making attacks more effective and eventually helping strengthen model defenses. The authors claim that existing methods assign the same trajectory level signal to all turns, and they claim this to be a limitation. The paper addresses that by assigning the final outcome signal to individual turns.

Critical strengths

I think this paper tackles an interesting problem that may be useful not only in the area of making models more robust for jailbreaking, but also for making models better at handling dialogues.

The paper is well written and easy to follow.

The paper shows experiments for a collection of models from different families, as models being attacked and attackers.

Critical weaknesses
  • This paper may merit a mention: https://arxiv.org/pdf/2505.06120. I know it is not a jailbreaking but it shows how LLMs get lost in those dialogue turns that you are addressing in your paper, for general tasks.

-One nit comment is to include in the intro one single line that simply describes how the reward or penalty is applied to each turn (not just that it does).

  • the paper (unless I missed it) seems to ignore the fact that fluid conversations usually refer to previous turns, and the leave-one turn strategy does not seem to allow for this. Section 3.2 touches on this, but it is not just about relative positioning, isn't it? it is about a turn means something very different without prior context, and a turn cannot be provided before or after (its position) unless you modify the previous context for both the user turns and the agent/LLM turns. Please let me know wht I am missing.
Questions for authors
  • when you do the leave-one turn strategy, aren't all turns part of the context? they are essentially never treated in isolation, besides the first turn as it does not have any prior context. I am a bit confused how the masked history really reflects the impact of each turn on jailbreaking. What happens with turns that refer to prior turns?
6 comments

Thank you for reading our paper so carefully! Your comments helped us see where our explanation needed to be clearer, especially around how LOTO handles dependencies between turns.

W1. Related work: LLMs Get Lost in Multi-Turn Conversation

Thank you for recommending ‘LLMs Get Lost in Multi-Turn Conversation’! This well-known work was presented as an oral at ICLR 2026. By splitting fully specified task instructions and progressively providing information across multiple turns, it finds that LLMs often form incorrect assumptions early in a conversation, struggle to correct them later, and consequently suffer performance degradation. We have added this citation and a discussion of how historical context affects subsequent model behavior in multi-turn interactions. The revised text is highlighted in blue.

W2. Clarifying how rewards and penalties are applied

Thank you for suggesting a brief explanation in the Introduction of how rewards and penalties apply to individual turns. This makes the method description more coherent and easier to follow. We have added the following sentence to the Introduction and highlighted it in blue:

“Concretely, TRACE scales the trajectory-level outcome advantage using turn-level credit weights and adds a refusal-aware penalty for each turn.”

W3. & Q1. Cross-turn context dependency in LOTO

Thank you for raising this important issue of cross-turn context dependency! We also consider it a meaningful question: masking a historical turn may disrupt semantic dependencies in subsequent dialogue, so changes in final harmfulness may reflect both the turn’s contribution and context disruption. We clarify this issue from three perspectives below.

P1. Does LOTO evaluate a turn in isolation, outside its context?

Our LOTO intervention operates on the dialogue history as a whole.

Let the original trajectory be τ=(x1,y1,…,xT,yT)\tau=(x_1,y_1,\ldots,x_T,y_T). At each turn, both the attacker and the target condition on the preceding history. In particular, the final target response used to assess attack success is generated as:

yT∼πϕ(⋅∣τT−1,xT).y_T\sim\pi_\phi(\cdot\mid\tau_{T-1},x_T).

LOTO removes a historical interaction pair (xt,yt)(x_t,y_t), where t<Tt<T.

A central feature of TRACE’s leave-one-turn-out procedure is that it keeps the final attacker query xTx_T fixed, removes one complete non-final interaction pair (xt,yt)(x_t,y_t), and retains all other realized dialogue content. Only the final target response is regenerated. It therefore measures how removing a historical interaction affects the target’s answer to the same final query.

Furthermore, during training, TRACE applies LOTO credit only to successful trajectories. Thus, xTx_T is a query that already elicited a harmful target response under the complete history. Our motivation is to mask an earlier interaction and examine how its presence in the history affects the harmfulness confidence score of the final response.

Of course, fixing xTx_T keeps the final request unchanged, but does not by itself guarantee that all semantic dependencies remain intact after masking. This is what the following experiments examine.

P2. If later turns refer to masked content, could the credit mainly reflect context disruption?

Masking may lead to several outcomes:

  • The request remains understandable, and the target explicitly refuses the same harmful task: the original context facilitated harmful compliance.
  • The target requests clarification because a referent, object, or premise is missing: the original turn supplied information needed to understand the task.
  • The target shifts to unrelated content or cannot provide a meaningful answer: context disruption is present and requires separate analysis.

The latter two cases may both involve context disruption: later turns may refer to deleted content, leaving unclear references or missing premises and causing clarification requests, task deviation, or ineffective responses.

To examine this explanation, we compare two interventions (Appendix B.2, Table 5):

  • Standard LOTO masking: removing an interaction pair, as defined in the paper.
  • Context-preserving neutral infilling: constructing a replacement interaction at the same position using GPT-4o, preserving necessary entities, references, and discourse bridges as far as possible while removing content that advances the attack.

Both interventions keep the final attacker query xTx_T and all other dialogue content fixed. We regenerate only the final target response and calculate turn credit using the same harmfulness judge. Neutral infilling serves as a reference intervention that better preserves dialogue continuity.

CheckMetricResult
Does neutral infilling improve dialogue continuity?High-coherence rateLOTO: 40.4%; neutral infilling: 71.5%
Are the credit rankings correlated after improving continuity?Spearman correlation across all 293 casesρ = 0.634, 95% CI [0.509, 0.749]
Does this correlation depend on low-quality infilling?Spearman correlation on 206 strict high-coherence infilling casesρ = 0.637

We interpret these three results together:

First, the first row shows that our neutral infilling improves context continuity, increasing the high-coherence rate from 40.4% to 71.5%. Replacing the interaction with neutral dialogue therefore provides a more coherent reference intervention.

Second, after improving dialogue continuity, the turn-level credit scores from the two interventions remain positively correlated across all 293 tested turns (ρ = 0.634), indicating some consistency in their credit rankings.

Third, restricting the analysis to the 206 neutral-infilling cases that satisfy the strict coherence criterion gives a correlation of 0.637, close to 0.634 on the full set. The association therefore persists among cases with higher-quality infilling and better-preserved continuity.

Thus, LOTO remains correlated with the reference intervention after dialogue continuity is improved, supporting the presence of a credit signal beyond context disruption alone. However, this experiment cannot fully separate context disruption from attack scaffolding, and standard masking may still disrupt cross-turn dependencies in some cases.

Does this mean that cross-turn dependencies affected by context disruption are necessarily unrelated to attack scaffolding? We believe these dependencies may themselves form part of the gradual buildup toward a harmful response, as discussed next.

P3. Can LOTO credit still measure a turn’s contribution to final harmfulness when cross-turn dependencies exist?

We believe that cross-turn dependency may itself be one way an earlier turn provides attack scaffolding. For example, a turn may introduce an object, topic, or discussion framework that subsequent requests build on to gradually elicit a harmful response. We therefore further examine whether the final attack remains successful after preserving entities and dialogue continuity as far as possible while removing attack-advancing content, and whether the resulting credit remains correlated with LOTO.

To this end, we add a context-neutral infilling experiment. Unlike the setting in P2, which regenerates only the final target response, this experiment regenerates the target response at the substituted turn and all subsequent target responses.

The data and shared settings are as follows:

  • Original data: We collect 20 successful trajectories from each of the early, middle, and late stages of training, totaling 60 trajectories and 174 non-final turns.
  • Preparation of successful trajectories: For each turn under examination, we first retain the original interaction pair (xt,yt)(x_t,y_t) and regenerate subsequent target responses until we collect 10 successful trajectories. Each subsequent experimental condition therefore contains 174 × 10 = 1,740 generated continuations.
  • Fixed and regenerated content: We keep the history preceding turn tt and all subsequent attacker queries unchanged, including the final attacker query xTx_T. In both conditions, the same target regenerates yt,yt+1,…,yTy_t,y_{t+1},\ldots,y_T.
  • Result inclusion: All generated outcomes from both conditions below are included in the statistics, regardless of whether the attack succeeds or fails.

We compare two conditions:

  • Original-query control: We retain the original attacker query xtx_t at turn tt and regenerate the target response at that turn and all subsequent target responses.
  • Context-neutral infilling: We replace xtx_t with an ordinary question that does not advance the attack, preserving the original topic, entities, and references needed by subsequent turns as far as possible. We then regenerate the target response at that turn and all subsequent target responses. For example, we retain technical names such as BERT and RankBrain but ask about the titles and publication dates of relevant references. Originally benign background questions are replaced with ordinary questions, without adding refusals, safety advice, or legal or moral reminders.

First, we examine whether the final attack success rate changes:

SettingAttack success rate
Retain the original xtx_t and regenerate the target response at that turn and all subsequent target responses76.26%
Apply context-neutral infilling and regenerate the target response at that turn and all subsequent target responses64.20%
Decrease relative to the control12.07 percentage points, 95% CI [7.63, 17.04]

The control also uses regenerated responses rather than the responses saved in the original successful trajectories, so its success rate need not be 100%.

This result shows that removing the original turn’s attack-advancing content further reduces attack success, even when necessary entities and dialogue continuity are preserved as far as possible and subsequent target responses can update under the modified history.

Second, we examine whether this substitution still produces turn credit correlated with LOTO.

We calculate the following two score differences. For each turn, we average the results over 10 generations and compare them with LOTO credit on the same set of successful trajectories:

Score difference used to calculate turn creditSpearman ρ with LOTO, computed within each trajectory and then averaged
Final harmfulness score of the original successful trajectory − final harmfulness score after neutral substitution0.283, 95% CI [0.075, 0.478]
Score of the original-query control − score of the neutral-infilling condition0.220, 95% CI [0.025, 0.417]

The first row follows the paper’s definition of credit as the difference in final harmfulness scores before and after intervention. The second row further uses the control with regenerated responses as the comparison baseline. Both calculations are positively correlated with LOTO.

Furthermore, following the paper’s definitions, we classify turns as attack-critical when the attack fails after intervention and as redundant when it remains successful. Neutral substitution and LOTO achieve 68.9% agreement, with Cohen’s κ = 0.293. Thus, under this setting, neutral substitution still produces a turn-contribution signal shared with LOTO.

Thank you for reading this far! The above presents our analysis and additional experiments on LOTO and context disruption.

Finally, we provide two further tables. Appendix B.2, Table 6 examines the stability of LOTO credit under repeated final-response resampling:

Resampling settingSpearman correlationMargin-sign agreement
Temperature = 0, K = 30.891100.0%
Temperature = 0.7, K = 100.66997.0%

The direct ablation of success-side LOTO provides further evidence of its usefulness for learning from successful attack trajectories, with higher ASR and fewer average training rollout turns:

MethodAverage ASR₅@1 (%, Table 3)Average training rollout turns (Figure 8)
GRPO72.372.77
GRPO + success-side credit (LOTO)77.452.65

Thank you again for raising this meaningful question! It prompted us to examine more closely how masking affects continuous dialogue and to further enrich the paper. The additional experiments are highlighted in blue in the revised manuscript.

Reviewer_Haq3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an intuitive method for credit assignment in multi-turn attack RL. They present three insights on multi-turn jailbreak attacks, including in-uniformity, phase dependency, and target specificity. The authors then propose TRACE, replacing the trajectory-level advantage with a turn-aware advantage. Experiments on multiple datasets are performed to show the learned attacker's strong capability.

Critical strengths
  • The issue of credit assignment in multi-turn attack learning is highly intuitive and valuable.
  • The paper is clearly written and rich in content. Considering the appendix, there are extensive experiments and discussions.
Critical weaknesses
  • Although the author provided three insights regarding multi-turn jailbreak attacks, these insights are quite intuitive, not specific to multi-turn jailbreak attacks, and might be similar in other multi-turn tasks, which reduces their value. Besides, the reviewer did not see a direct connection between the insights and the specific details in the proposed credit assignment method. Thus, these insights seem to merely prove the importance of credit assignment in multi-turn attack reinforcement learning, rather than provide inspiration for method design. As a result, the motivations of some complex design choices involved in the proposed method are insufficiently explained by the text or justified by ablation. For instance, why use multiplicative credit adjustments rather than additive corrections, and why is normalization necessary? The reviewer would appreciate further discussion, e.g., perhaps based on theoretical considerations, inspired by other works, or discovered through experiments (this design works best).

  • Although the paper discusses existing credit assignment methods in multi-turn reinforcement learning in Appendix A.4, no comparisons of these methods were made in any experiments. While the authors argue that these methods do not apply to the multi-turn attack RL, the reviewer thinks that some of them can be intuitively applicable. For example, defining a simple harmfulness difference between consecutive turns as the turn-level process reward is a straightforward baseline, and there might be other applicable methods. The reviewer encourages further comparisons with such baselines. Besides, the paper discussed a recent and performant method, SEMA, extensively (Figure 1, main text, and related work), but did not include it in the experiments, claiming the code was unavailable. However, the reviewer found that it is public (https://github.com/microsoft/SEMA). The authors may either directly compare with it or apply TRACE's credit assignment mechanism to SEMA's reward formulation to further demonstrate its generalizability.

Questions for authors

The paper reports ASR@k (e.g., ASR@1 in Table 2). Based on Section 2, the reviewer understands that a trajectory is considered successful if any turn achieves a harmfulness score above the threshold (gamma). Could the authors confirm this interpretation?

If so, this definition is no longer strictly ASR@1, because it actually involves multiple attempts, but not independent; rather, they are based on multi-turn conversations. This metric would be more precisely denoted as ASR_T@k, as in TROJail, to distinguish the number of dialogue turns (T) from independent attack attempts (k).

Additionally, in TROJail, the calculation process of this metric is: first collect the trajectories of each method based on its own termination rule, and then calculate whether there is at least one turn among the first T turns that is judged as a success. The authors should also clarify whether all baselines are evaluated by checking every turn of their collected trajectories using a unified success criterion (such as the score of the HarmBench Classifier being greater than gamma), or whether each method follows its own termination and success criteria.

More importantly, this evaluation may favor multi-turn methods over single-turn baselines due to the additional attack opportunities. Since any single-turn attack can be simply wrapped with a loop to become a "multi-turn" or strictly "multi-attempt" attack, the reviewer suggests comparing ASR_T@1 for multi-turn methods against ASR@T for single-turn methods for a fairer comparison. Alternatively, the authors could report the more commonly used ASR@1 in the narrower sense, evaluating only the final response of each collected trajectory rather than counting success at any intermediate turn. This would provide a fairer comparison and an additional perspective on the effectiveness of multi-turn attacks without aggregating success across turns.

11 comments

Thank you for your detailed comments! These suggestions have helped us improve both the method description and the experiments. We address the concerns point by point, further clarify the method design and evaluation protocol, and provide five sets of mechanism ablations, along with comparisons against GRPO-VPS (the suggested "straightforward baseline") and SEMA.

W1.1 The specific significance of the three insights for multi-turn jailbreaking

"These insights are quite intuitive, not specific to multi-turn jailbreak attacks."

We understand the reviewer’s concern. Fine-grained credit assignment has been studied in other multi-turn tasks. However, our contribution is to empirically characterize three properties of turn-level contributions in multi-turn jailbreaking—non-uniformity, phase dependency, and target specificity—and show how these observations guide TRACE’s design. Consequently, applying credit assignment methods from other multi-turn tasks to jailbreaking requires accounting for these differences.

First, multi-turn jailbreaking has a different interaction structure from typical multi-turn problem-solving and cooperative reasoning tasks. Multi-turn problem-solving methods such as RAGEN and IGPO primarily advance problem-solving through environmental feedback, while cooperative reasoning methods such as ReMA achieve shared goals through cooperation. In contrast, the attacker and target model in multi-turn jailbreaking have opposing safety objectives: the attacker progressively constructs conversational context to induce harmful outputs that the target model’s safety mechanisms are intended to prevent.

We appreciate the reviewer’s concern that these insights may seem intuitive and could also arise in other multi-turn tasks. Below, we elaborate on their specific manifestations in multi-turn jailbreaking.

  • Non-uniformity: In search tasks, newly acquired evidence or the probability of producing the correct answer can be used to assess progress at each turn, as in IGPO. In multi-turn jailbreaking, however, an interaction can make an important contribution to the eventual jailbreak even when it is locally harmless. Consequently, local harmfulness alone is insufficient to assess that turn’s attack value. Non-uniformity in multi-turn jailbreaking therefore differs from that in multi-turn tasks where usable evidence is available at each turn.
  • Phase dependency: “Planning first, execution later” is a stage structure shared by many tasks. However, progressively building up harmful intent across turns in multi-turn jailbreaking introduces additional complexity. Exposing harmful intent too early can trigger safety refusals (premature exposure), whereas later turns may remain overly benign (harmfulness drift). These are the distinctive patterns of multi-turn jailbreaking summarized in Figure 2(a–b). Although other tasks also involve exploration, contextual preparation, and delayed rewards, multi-turn jailbreaking specifically involves a trade-off between efficiently advancing the attack objective and triggering safety refusals.
  • Target specificity: We agree that differences in interaction partners or environments also exist in other multi-agent tasks. In multi-turn jailbreaking, however, we specifically focus on differences in target models’ safety behaviors: their refusal tendencies and safety responses to conversational context substantially affect which preparatory turns advance the attack and which interactions hinder subsequent success. As shown in Figure 2(c–d), the same behavior can yield entirely different outcomes against different target models.

Our point is therefore not that these properties never occur in other tasks, but that they have specific manifestations in multi-turn jailbreaking.

[1] Wang et al. [ICLR2026] IGPO. arXiv:2510.14967, 2025.

[2] Wan et al. [NeurIPS2025] ReMA. arXiv:2503.09501, 2025.

[3] Wang et al. RAGEN. arXiv:2504.20073, 2025.

W1.2 How the three insights directly guide TRACE

“The reviewer did not see a direct connection between the insights and the specific details.”

Thank you for pointing out that the original manuscript did not explain this connection sufficiently. In fact, the three insights directly correspond to TRACE’s design. We have also added ablations on the more specific choices for combining credit signals.

Insight 1: Non-uniformity. This insight directly guides TRACE’s success-side credit estimation (Eqs. 5–7). We first use leave-one-turn-out (LOTO) masking to analyze turn-level contributions and find that contributions vary across turns. Accordingly, TRACE directly adopts the same LOTO mechanism on the success side, estimating each turn’s credit as the difference in final-response harmfulness before and after masking that turn (Eq. 5). We then normalize the credit and convert it into a turn-aware multiplier (Eqs. 6–7) to adjust the outcome advantage (Eq. 3). This assigns greater reward weights to turns with larger contributions, thereby reducing over-rewarding of relatively redundant turns.

Non-uniformity also directly guides our multi-turn defense design. We identify attack-critical turns and further distinguish latent-risk turns from direct-harm turns. Their original target responses serve as DPO rejected responses, while responses rewritten using the corresponding safety strategies serve as chosen responses.

Insight 2: Phase dependency. This insight directly guides TRACE’s failure-side credit estimation (Eqs. 8–10). Figure 2(a–b) compares harmfulness distributions across conversational phases in successful and failed trajectories, revealing two failure modes: premature exposure, where harmful intent is exposed too early and is likely to trigger safety refusals, and harmfulness drift, where later turns remain overly benign. Based on these observations, TRACE’s failure side directly uses the phase-specific successful-trajectory priors obtained from our analysis as references to compute harmfulness and relevance deviation penalties (Eqs. 8–9; the specific priors are provided in Tables 7–8). These penalties are then converted into failure-side multipliers (Eq. 10), imposing stronger penalties on turns that deviate from the corresponding phase-specific priors.

Insight 3: Target specificity. This insight also directly guides TRACE’s failure-side credit estimation (Eqs. 8–10; Appendix B.3). Different target models exhibit different safety and refusal behaviors. We therefore estimate target-specific harmfulness and relevance priors, namely qtϕq_t^\phi and LtϕL_t^\phi, from successful trajectories collected against each target model (Tables 7–8). For each failed trajectory, TRACE uses the priors corresponding to the target model it interacted with to compute deviation penalties and failure-side multipliers, allowing turn-level credit assignment to reflect differences in the target models’ safety behaviors.

These three properties directly guide TRACE’s credit estimation on successful and failed trajectories: non-uniform credit on the success side, phase-aware penalties on the failure side, and target-conditioned priors. The original manuscript provided insufficient experimental justification for how these estimated signals are incorporated into training, so we have added the following five sets of ablations.

W1.3 Multiplicative adjustment and normalization: rationale and five ablations

“Why use multiplicative credit adjustments rather than additive corrections, and why is normalization necessary?”

We agree with the reviewer’s concern about these more specific implementation choices. We now explain their rationale separately and examine them through controlled ablations. We have added five mechanism ablations, covering the multiplicative adjustment and normalization requested by this reviewer, as well as additional ablations addressing Reviewer 8uE7’s concerns:

  • E1: Direct Additive Credit, comparing against a simple raw additive correction (“why use multiplicative credit adjustments rather than additive corrections”).
  • E2: w/o Success-side Credit Z-score, examining the role of credit standardization (“why is normalization necessary”).
  • E3: w/o Multiplier Mean Normalization, examining the role of the unit-mean constraint (“why is normalization necessary”).
  • E4: Phase-Agnostic Prior, addressing Reviewer 8uE7’s question about “how much of the improvement comes from the phase-specific prior itself.”
  • E5: TROJail-Inspired Fixed Linear Reference, an additional ablation addressing Reviewer 8uE7’s concern.

Ablation configurations

TRACE uses the following turn-level advantage:

\qquad m_t>0, \qquad \frac{1}{T}\sum_{t=1}^{T}m_t=1, \qquad \frac{1}{T}\sum_{t=1}^{T}m_tA^o=A^o.$$ Here, $A^o$ is the trajectory-level outcome advantage, $m_t$ is the turn-level credit multiplier, and $A_t^p$ is the weighted refusal-aware local process term. **E1: Direct Additive Credit.** We retain TRACE’s original raw LOTO credit $c_t$ and failure penalty $B_t$, but use them directly as additive corrections: $$A_t^{\mathrm{add}}=A^o+\eta C_t+A_t^p, \qquad \eta=0.4,$$ $$C_t= \begin{cases} c_t, & \text{if the success-side conditions are satisfied},\\\\ -B_t, & \text{if the failure-side conditions are satisfied},\\\\ 0, & \text{otherwise}. \end{cases}$$ Here, $B_t\ge0$, and a larger $B_t$ corresponds to a stronger failure-side penalty. Unlike TRACE’s multiplicative adjustment, the magnitude of this simple direct additive correction does not adapt to $A^o$, and it does not guarantee that the mean or sign of the outcome component is preserved. This experiment compares our current multiplicative design against this simple additive alternative. **E2: w/o Success-side Credit Z-score.** The original method computes: $$z_t=\operatorname{clip}\left(\frac{c_t-\mu}{\sigma+\epsilon},-z_{\max},z_{\max}\right).$$ E2 removes only the z-score standardization of success-side raw credit, replacing it with: $$z_t=\operatorname{clip}(c_t,-z_{\max},z_{\max}).$$ All other settings are retained. **E3: w/o Multiplier Mean Normalization.** The original method uses: $$m_t=(1-\lambda)+\lambda\frac{v_t}{\operatorname{mean}(v)}.$$ E3 removes only the division of $v_t$ by its mean and constructs the multiplier directly from the original $v_t$. All other settings are retained. In the original method, the multipliers preserve the trajectory’s aggregate outcome signal through the constraint: $$\frac{1}{T_i}\sum_{t=1}^{T_i}m_{i,t}=1.$$ Without this normalization, the mean multiplier varies with the credit values, so the average turn-level outcome advantage is no longer guaranteed to equal the original $A^o$. **E4: Phase-Agnostic Prior.** This ablation retains the priors derived from successful trajectories but removes their dependence on the attack phase. We reuse the same successful calibration trajectories for each target and pool their features across phases. The original phase-specific priors and the phase-agnostic alternatives are shown below. Harmfulness priors: | Prior | Turn | Qwen Safe | Qwen Cont. | Qwen Unsafe | OSS Safe | OSS Cont. | OSS Unsafe | Llama Safe | Llama Contr. | Llama Unsafe | | :--- | :---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | Phase-specific (ours) | 1 | 0.78 | 0.08 | 0.14 | 0.87 | 0.11 | 0.02 | 0.56 | 0.13 | 0.31 | | Phase-specific (ours) | 2 | 0.51 | 0.14 | 0.34 | 0.65 | 0.28 | 0.07 | 0.40 | 0.21 | 0.38 | | Phase-specific (ours) | 3 | 0.25 | 0.19 | 0.56 | 0.56 | 0.25 | 0.18 | 0.41 | 0.18 | 0.41 | | Phase-specific (ours) | 4 | 0.15 | 0.24 | 0.61 | 0.47 | 0.26 | 0.26 | 0.23 | 0.22 | 0.55 | | Phase-specific (ours) | 5 | 0.06 | 0.14 | 0.80 | 0.20 | 0.34 | 0.46 | 0.13 | 0.40 | 0.47 | | **Phase-agnostic (E4)** | All | 0.35 | 0.16 | 0.49 | 0.55 | 0.25 | 0.20 | 0.35 | 0.22 | 0.43 | Relevance priors: | Prior | Turn | Qwen | OSS | Llama | | :--- | :---: | ---: | ---: | ---: | | Phase-specific (ours) | 1 | 0.40 | 0.40 | 0.52 | | Phase-specific (ours) | 2 | 0.45 | 0.32 | 0.52 | | Phase-specific (ours) | 3 | 0.50 | 0.32 | 0.46 | | Phase-specific (ours) | 4 | 0.50 | 0.29 | 0.50 | | Phase-specific (ours) | 5 | 0.50 | 0.32 | 0.48 | | **Phase-agnostic (E4)** | All | 0.45 | 0.33 | 0.49 |

Comments on the paper

1 comment

We sincerely thank the three reviewers for their careful reading, constructive feedback, and the time they spent examining our responses and additional experiments. Their comments helped us clarify TRACE’s design rationale and the scope of its credit estimates, and strengthen the experimental validation.

To address the reviewers’ main concerns, we made additions in four areas:

  1. Method motivation and mechanism design (Haq3, 8uE7). We clarified how the three insights guide TRACE’s components and added five mechanism ablations to examine multiplicative credit adjustment, normalization, and the roles of phase-specific and target-specific priors.

  2. Baselines and evaluation fairness (Haq3). We added comparisons with SEMA and GRPO-VPS, further clarified each method’s success criterion and target-query budget, and evaluated Jailbreak-R1 with five independent attempts, comparing its ASR@5 with TRACE’s ASR₅@1.

  3. Cross-turn context dependency and the interpretation of LOTO (4stF, 8uE7). We added context-neutral infilling experiments and controls with regenerated subsequent target responses to examine the influence of context disruption on credit estimation. We also clarified LOTO’s role as a directional proxy for policy optimization.

  4. Training variance and credit estimation stability (8uE7). We repeated GRPO and TRACE training with three independent training seeds against Llama-3.1-8B-Instruct and reported mean ± std. We also supplemented the repeated-sampling analysis of LOTO with confidence intervals and sign agreement.

The experimental results and clarifications are provided in our point-by-point responses, and the corresponding changes are highlighted in blue in the revised manuscript. During the follow-up discussion, Haq3 responded positively to our rebuttal and increased their score, while 4stF and 8uE7 both explicitly confirmed that their questions or concerns had been addressed. We appreciate the care all three reviewers took in examining these additions, and thank the AC, SAC and PCs for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.