- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the credit assignment problem in RL-based multi-turn jailbreaking and proposes TRACE, a turn-aware framework that redistributes trajectory-level outcome rewards across dialogue turns. The authors empirically identify three properties of turn-level contributions—non-uniformity, phase dependency, and target specificity—and design asymmetric credit mechanisms for successful and failed trajectories. Successful trajectories use leave-one-turn-out semantic masking, while failed trajectories use phase-aware harmfulness and semantic relevance deviations, together with a refusal-aware local penalty. Extensive experiments show substantial improvements over prior multi-turn jailbreak baselines in attack effectiveness, transferability, and inference efficiency. The extracted credit signal is additionally reused for defense alignment.
Critical strengths
- The paper identifies a clear and consequential credit-assignment failure mode in existing multi-turn RL training.
- The experimental validation is broad and the gains are substantial.
- The paper goes beyond attack performance and shows that the same turn-level signal can be useful for defense
Critical weaknesses
- The turn-level credit estimator is intervention-dependent rather than a causal decomposition of trajectory return.
- The paper does not fully characterize training variance.
- The failure-side component is more heuristic than the success-side one.
Questions for authors
- For failed trajectories, how much of the improvement comes from the phase-specific prior itself? A comparison with a simpler phase-agnostic harmfulness/relevance reward would help clarify this.
- For the success-side credit, what happens if you regenerate the later dialogue after removing a turn, instead of keeping the remaining suffix fixed? Do you get similar turn rankings?
- Most experiments use a 5-turn budget. Do you have any results on longer conversations, and how does the extra leave-one-turn-out computation scale with the number of turns?
7 comments
Q1: Ablation for Phase-Agnostic Prior: replacing phase-specific priors with pooled priors
Thank you for this suggestion. It helps us distinguish the role of information derived from successful trajectories from that of phase-specific modeling, making our experimental analysis of the failure-side component more complete. We added two ablations:
- A1: Phase-Agnostic Prior, directly addressing the contribution of phase-specific modeling.
- A2: TROJail-Inspired Fixed Heuristic, an additional ablation that replaces the success-derived priors with fixed heuristic references.
Ablation configurations
A1: Pooled Phase-Agnostic Prior
In A1, we retain the prior information derived from successful trajectories but remove its phase dependence. All phases within a trajectory therefore use the same feature references. We keep the other conditions unchanged, reuse the same successful calibration trajectories for each target, and pool the features across phases rather than estimating separate priors for each phase.
The original phase-specific priors and the pooled phase-agnostic priors are shown below.
Harmfulness priors
| Prior | Turn | Qwen Safe | Qwen Cont. | Qwen Unsafe | OSS Safe | OSS Cont. | OSS Unsafe | Llama Safe | Llama Cont. | Llama Unsafe |
|---|---|---|---|---|---|---|---|---|---|---|
| Phase-specific (ours) | 1 | 0.78 | 0.08 | 0.14 | 0.87 | 0.11 | 0.02 | 0.56 | 0.13 | 0.31 |
| Phase-specific (ours) | 2 | 0.51 | 0.14 | 0.34 | 0.65 | 0.28 | 0.07 | 0.40 | 0.21 | 0.38 |
| Phase-specific (ours) | 3 | 0.25 | 0.19 | 0.56 | 0.56 | 0.25 | 0.18 | 0.41 | 0.18 | 0.41 |
| Phase-specific (ours) | 4 | 0.15 | 0.24 | 0.61 | 0.47 | 0.26 | 0.26 | 0.23 | 0.22 | 0.55 |
| Phase-specific (ours) | 5 | 0.06 | 0.14 | 0.80 | 0.20 | 0.34 | 0.46 | 0.13 | 0.40 | 0.47 |
| Phase-agnostic (A1) | All | 0.35 | 0.16 | 0.49 | 0.55 | 0.25 | 0.20 | 0.35 | 0.22 | 0.43 |
Relevance references
| Prior | Turn | Qwen | OSS | Llama |
|---|---|---|---|---|
| Phase-specific (ours) | 1 | 0.40 | 0.40 | 0.52 |
| Phase-specific (ours) | 2 | 0.45 | 0.32 | 0.52 |
| Phase-specific (ours) | 3 | 0.50 | 0.32 | 0.46 |
| Phase-specific (ours) | 4 | 0.50 | 0.29 | 0.50 |
| Phase-specific (ours) | 5 | 0.50 | 0.32 | 0.48 |
| Phase-agnostic (A1) | All | 0.45 | 0.33 | 0.49 |
Intuitively, pooling features across phases makes all turns share the same harmfulness and relevance references. This removes differences between the phase-specific reference distributions and their explicit modeling of turn-by-turn progression. The resulting reward applies the same criteria across phases and cannot express how the appropriateness of the same harmfulness level may differ by phase.
A2: TROJail-Inspired Fixed Heuristic
We added this experiment to further assess the need for TRACE's target- and phase-specific success priors. We remove the target-specific priors used for failed trajectories, so the references no longer depend on successful trajectories against the particular target.
Inspired by the fixed heuristic in TROJail §4.2–4.3,
we assign a fixed linear reference to each phase. For example, the relevance weights for a four-turn trajectory are 0.25, 0.50, 0.75, and 1.00. Harmfulness also uses a fixed increasing reference: 0.2, 0.4, 0.6, 0.8, and 1.0 for five-turn trajectories, and 0.25, 0.50, 0.75, and 1.00 for four-turn trajectories. Attacker-prompt harmfulness labels are mapped to safe = 0, controversial = 0.5, and unsafe = 1.
However, the prior tables above show that successful trajectories do not necessarily require increasingly related queries in later phases. Instead, successful attacks follow different pacing against targets with different refusal boundaries.
Ablation results
| Method | Qwen / HB | Qwen / JB | Qwen / WJB | Llama-3.1-8B-IT / HB | Llama / JB | Llama / WJB | GPT-OSS-20B / HB | OSS / JB | OSS / WJB | Average |
|---|---|---|---|---|---|---|---|---|---|---|
| GRPO | 81.55 | 77.58 | 81.33 | 71.48 | 74.55 | 69.67 | 64.15 | 66.06 | 65.00 | 72.37 |
| A1: Pooled phase-agnostic prior | 85.53 | 87.27 | 86.00 | 74.84 | 72.73 | 70.50 | 69.18 | 61.82 | 58.00 | 73.99 |
| A2: TROJail-inspired fixed linear reference | 87.42 | 76.36 | 87.00 | 76.10 | 74.55 | 79.50 | 51.57 | 52.73 | 45.50 | 70.08 |
| TRACE (single) | 87.84 | 95.15 | 89.83 | 79.66 | 84.24 | 84.00 | 82.80 | 76.97 | 84.17 | 84.96 |
| TRACE (mix) | 90.57 | 87.72 | 90.50 | 84.48 | 89.09 | 88.67 | 83.64 | 86.06 | 83.17 | 87.10 |
Replacing the phase-specific prior with the pooled phase-agnostic prior reduces the average ASR from 84.96% to 73.99%. One possible explanation is that a shared prior cannot adequately capture the different harmfulness and relevance requirements across phases, weakening the failure-side credit's ability to distinguish useful scaffolding from inappropriate behavior.
A2 achieves an average ASR of 70.08%, below GRPO's 72.37%. Although the success-side credit remains unchanged, replacing the failure-side design still reduces overall performance. This suggests that a fixed heuristic that does not reflect successful behavior against the specific target may struggle to accommodate target-dependent attack pacing. The current results therefore support TRACE's default design over the fixed heuristic tested in this ablation.
Q2: Evaluating LOTO credit using regenerated dialogue continuations and context-neutral infilling.
Thank you for this suggestion. We added an experiment to examine whether the turn rankings obtained by LOTO depend on the subsequent target responses already generated in the original trajectory.
First, we fix the subsequent attacker queries, including the final attacker query , and compare the following two settings:
- Original LOTO: Remove , retain the subsequently generated interaction pairs, and regenerate only the final response .
- New experiment: Remove , fix the subsequent attacker queries, and regenerate all subsequent target responses .
The new experiment therefore allows subsequent target responses to adapt to the modified history. We conduct it only on successful trajectories, consistent with TRACE's use of success-side credit.
We select 20 successful trajectories from each of the early, middle, and late stages of training, yielding 60 trajectories and 174 non-final turns. For each turn under examination, we first regenerate the subsequent target responses without removing any content and retain 10 successful trajectory realizations, yielding successful realizations in total.
We then apply both settings to the same successful realization. All outcomes after turn removal are retained, regardless of whether the attack remains successful. Both settings use the same harmfulness judge to compute credit. We average the 10 credit estimates for each turn and then compare the turn rankings within each trajectory.
| Evaluation | Result |
|---|---|
| Spearman ρ between the two turn rankings, averaged over 60 trajectories | 0.522, 95% CI [0.343, 0.675] |
| Agreement in attack-critical / redundant classifications | 80.2%, 95% CI [76.8%, 84.0%] |
We follow the definitions in the paper: for a successful trajectory, a turn is classified as attack-critical if removing it causes the attack to fail, and redundant if the attack remains successful.
Among cases classified as attack-critical by LOTO, 76.5% remain attack-critical after regenerating the subsequent target responses. Among cases classified as redundant by LOTO, 81.7% remain successful after regeneration. Thus, allowing subsequent target responses to update preserves similar turn rankings and the distinction between attack-critical and redundant turns.
Context-neutral infilling with target-response regeneration
We additionally conduct a context-neutral infilling experiment.
Following the principles in Appendix B.2, we preserve the original turn's entities and necessary references as much as possible while removing content that advances the attack. For example, we retain names such as BERT and RankBrain but rewrite the query to ask for the titles and publication dates of related papers. In a trajectory concerning explosive manufacturing, we replace an early scaffolding turn about a chemistry course with a question about a mathematics course. Ordinary background questions are rewritten as ordinary questions, without adding refusals or safety advice.
Both conditions regenerate all target responses from turn onward:
- Original-query replay: Retain the original attacker query at turn .
- Context-neutral infilling: Replace with an ordinary question that does not advance the attack while preserving contextual continuity.
All other attacker queries, including , remain fixed. The difference between the two conditions therefore helps assess whether the attack-progressing content in the original turn continues to contribute.
| Setting / comparison | Attack success rate / difference |
|---|---|
| Original-query replay | 76.26% |
| Context-neutral infilling | 64.20% |
| Reduction relative to the replay control | 12.07 percentage points, 95% CI [7.63, 17.04] |
We thank the reviewer for the thoughtful comments and constructive questions. These have motivated additional experiments that provide a more complete evaluation of TRACE and strengthen the empirical support for our conclusions.
W1: The turn-level credit estimator is intervention-dependent rather than a causal decomposition of trajectory return.
We agree with the reviewer's assessment. TRACE's success-side credit is an intervention-dependent directional proxy for turn-level credit assignment.
In multi-turn jailbreaking, the judge provides a trajectory-level harmfulness score: it evaluates whether the final target response fulfills the original harmful request, but does not provide ground-truth labels for each turn's contribution to the outcome. Different turns may depend on one another, jointly establish context, and exert delayed effects. Therefore, the final attack outcome alone does not uniquely determine how credit should be distributed across turns. This is one reason why credit assignment in multi-turn jailbreaking is challenging.
Other credit assignment methods define their signals according to their respective objectives. For example, IGPO defines turn-level rewards through changes in the model's probability of generating a known correct answer, providing a computable progress signal for multi-turn search. SCAR first defines a utility function over text subsets and then uses Shapley values to assign token- or span-level credit. These methods provide credit under their respective definitions, but do not directly yield a unique attribution of final jailbreak success to individual turns.
TRACE fixes the final attacker query xT, masks a non-final interaction pair, and regenerates the final target response to compute credit from the harmfulness-score change. This intervention-relative sensitivity guides turn-level training feedback (Appendix B.1). We evaluate its stability through intervention and repeated-resampling experiments (Appendix B.2), and its training utility through ablations (Table 3). We will clarify this interpretation in the main text.
[1]. Wang et al. IGPO ICLR 2026; arXiv:2510.14967, 2025.
[2]. Cao et al. SCAR arXiv:2505.20417, 2025.
W2: The paper does not fully characterize training variance.
Thank you for this suggestion. We have added experiments to characterize training variance. Using Llama-3.1-8B-Instruct as the target, we independently trained GRPO and TRACE (single) with three training seeds: 10000, 20000, and 30000. The results are reported below as ASR₅@1 (%).
TRACE (single) achieves a higher Macro ASR than GRPO under all three training seeds, with an average improvement of 4.70 percentage points and a lower cross-seed standard deviation (1.49 vs. 3.59). These results indicate that its performance gains are relatively stable across these three independent training runs.
In addition, Table 6 in Appendix B.2 provides further analysis of the sampling variability of the LOTO credit used during training.
Under the temperature = 0 setting used in training, the credit estimates exhibit high ranking consistency under repeated resampling. With greater sampling randomness, the rankings remain positively correlated, and the signs of credits away from zero also remain highly consistent.
W3: The failure-side component is more heuristic than the success-side one.
We agree that the failure-side turn-aware deviation penalty has a stronger heuristic component than the success-side credit. Its design is motivated by the premature exposure and harmfulness drift observed in Section 3.2, together with the target specificity discussed in Section 3.3.
Specifically, we estimate harmfulness and relevance priors from successful trajectories against the corresponding target, and then assign penalties according to how much each turn in a failed trajectory deviates from these priors (Appendix B.3). This component is therefore an empirically motivated penalty design for policy optimization.