acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Trajectory Return Attribution and Contrastive Erasure for Multi-Turn Safety

Abstract

Safety-aligned large language models (LLMs) often refuse a harmful request but comply once the same goal is spread over several turns. Preference objectives score whole responses to single prompts, so their training loss alone cannot control risk on unseen histories. Our analysis gives sufficient conditions under which suppression at supervised single-turn contexts yields a bound on multi-turn trajectory risk. The bound accounts for coverage, transfer slack, and leakage, and characterizes contraction relative to a base-policy risk budget evaluated on the trained policy’s contexts. **TRACE** *(Trajectory Return Attribution and Contrastive Erasure)* turns this principle into a token-level objective. On the safe response, each token is weighted by the discounted return of a refusal-attributable advantage. The advantage compares a frozen reference model with its refusal-ablated copy, allowing earlier response tokens to receive credit from later refusal-related evidence. At high-gap positions on rejected responses, **TRACE** combines the observed token with policy-selected alternatives in the erasure target. A gradient-norm penalty replaces the retain set. Across five open-weight models and seven multi-turn attacks, **TRACE** gives the lowest attack success rate (ASR) in all 35 model and attack pairs, while the model utility evaluated on MMLU and HellaSwag drop by at most 1.23 points. Source code can be found in the supplemental material.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-turn Robustness, Defending Against Attacks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nCiYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tries to improve multi-turn safety using single-turn preference supervision. Authors propose TRACE and develop a solid formulation to address multi-turn trajectory risk.

Critical strengths

1. The paper studies a practically meaningful problem and show that TRACE can improve multi-turn safety. Framing the safety problem under interaction histories is insightful and reasonable.

2. The overall method is solid and the paper is well organized and written. In particular, the roles of TAW, RLPE, and gradient-norm regularization are clearly motivated and analyzed.

Critical weaknesses

1. The single-turn to multi-turn transfer seems to rely heavily on how well the supervised contexts match the visited contexts during attack trajectories. I notice that the theory tries to capture this gap. However, the training context is a single-turn harmful request and the deployment context contains the full multi-turn history. I am not clear how stable this correspondence is in practice, or whether the resulting context shift can indeed be summarized by a small transfer slack. The current experiments mainly report downstream ASR and lack analysis of relevant quantities.

2. RLPE seems to rely on policy-ranked high-probability alternatives as proxies for the unsafe mass, without an explicit semantic criterion. This paper describes the likelihood gap as a contrastive selection signal, not a harmfulness label, and that proxy-token suppression depends on the current policy mass and the mixed target. I am therefore unsure why these alternatives should reliably represent the unsafe choices that need to be suppressed, especially when some of them may be benign. Theorem 4.4 connects this mechanism to trajectory risk through proxy coverage and additional tokenwise suppression conditions, but these conditions appear to be assumed instead of being implied by the RLPE objective.

Questions for authors

please see the weaknesses above. In particular:

1. How well do the supervised contexts match the contexts actually visited during multi-turn attacks? Can you report coverage or transfer slack beyond final ASR?

2. How often do RLPE’s policy-ranked proxy tokens actually overlap with unsafe alternatives, and how often do they include benign ones?

3. Theorem 4.4 assumes unsafe proxy tokens are suppressed and the remaining unsafe tokens do not increase. How often do these conditions hold after training?

4. kproxyk_{\mathrm{proxy}} changes position selection, proxy support, and loss normalization at the same time. Can you separate these effects to show what is driving the gain?

Reviewer_6RWHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how token-level supervision on single-turn preference data can improve safety against multi-turn attacks. The authors argue that refusal-related evidence may become more pronounced after a safe response has begun, while suppressing a rejected response may leave alternative harmful continuations available. For safe responses, TRACE constructs token-level signals using a reference model and its refusal-ablated counterpart, then accumulates later signals backward to earlier tokens. For rejected responses, it uses reference-normalized log-probability gaps to select positions and constructs erasure targets containing both the observed tokens and high-probability alternatives.

Critical strengths

Appropriately attributing safety-related signals is an important and challenging problem. I find the idea of allowing later refusal-related evidence to contribute to the supervision of earlier response tokens intuitive and particularly inspiring. On the rejected-response branch, the authors also explore position selection and target-construction refinements relative to the related BS-T formulation. The central idea of backward accumulation is the aspect of this work that I find most insightful.

Critical weaknesses
  1. Since rjr_j can be positive or negative, its discounted accumulation in GjG_j can involve cancellation. Although the subsequent normalization and mapping keep the final weights positive, it remains unclear whether such cancellation can underweight important refusal-related positions.

  2. The use of rjr_j as a refusal-related signal would benefit from further validation. Although the ablated scorer is constructed by removing a refusal direction from the reference model, this intervention may also affect other aspects of generation. It is therefore unclear how reliably the resulting log-probability difference isolates refusal-related contributions.

  3. Table 13 evaluates over-refusal only on Llama-3.1-8B. Under the default configuration, the refusal rate on safe XSTest prompts increases from 5.48% to 17.23%. The authors acknowledge this trade-off and report a better balance with a smaller regularization radius. However, the results do not establish whether this trade-off generalizes to other models or whether backward accumulation itself contributes to the increased refusal rate.

  4. Appendix F.4 explicitly states that the W-DOOR configuration in Tables 2 and 9 uses constant token weights and omits the retain loss. Although this limitation is disclosed, these experiments do not establish an advantage over the full weighted, retain-set version of W-DOOR.

Questions for authors

see the weaknesses above.

Reviewer_86vQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem that LLMs aligned with single-turn safety data can still be vulnerable to multi-turn jailbreak attacks. It proposes TRACE, which improves multi-turn safety using only single-turn preference data. TRACE includes trajectory-aware adaptive weighting to assign token weights based on future refusal-related evidence, risk-localized proxy erasure to suppress harmful responses and likely alternatives, and a gradient-norm penalty to reduce model drift. Experiments on five open-weight LLMs and seven multi-turn attacks show that TRACE consistently reduces attack success rates while largely preserving general model utility.

Critical strengths
  • The paper studies an important problem in LLM safety, namely the gap between single-turn safety alignment and multi-turn robustness.
  • The experiments are extensive, covering five LLMs and seven multi-turn attacks, and TRACE consistently outperforms the evaluated baselines.
Critical weaknesses
  1. The theoretical analysis relies on several strong assumptions. In particular, Theorem 4.4 assumes that the unsafe proxy tokens are suppressed and that the remaining unsafe tokens do not have higher probabilities than those under the base model. However, the RLPE objective itself does not guarantee these conditions, and the paper does not verify whether they hold after training.
  2. The token selection in RLPE is based on the reference-normalized likelihood gap rather than the actual harmfulness of the token. Therefore, the selected positions may contain benign or common tokens, while some truly harmful tokens may not be selected if their likelihood gap is small. It would be useful to analyze whether the selected tokens are indeed related to harmful content. The paper itself also states that this gap is a contrastive signal rather than a semantic harmfulness label.
  3. The proposed method is closely related to the refusal behavior of the model, but the evaluation does not include strong attacks that directly target refusal. Recent methods such as TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Models and Don’t Say No: Jailbreaking LLM by Suppressing Refusal explicitly suppress refusal behavior and have shown strong attack performance. It would be useful to evaluate whether TRACE can still maintain its safety gains under these refusal-targeted attacks.
Questions for authors

Please see the Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.