- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper tries to improve multi-turn safety using single-turn preference supervision. Authors propose TRACE and develop a solid formulation to address multi-turn trajectory risk.
Critical strengths
1. The paper studies a practically meaningful problem and show that TRACE can improve multi-turn safety. Framing the safety problem under interaction histories is insightful and reasonable.
2. The overall method is solid and the paper is well organized and written. In particular, the roles of TAW, RLPE, and gradient-norm regularization are clearly motivated and analyzed.
Critical weaknesses
1. The single-turn to multi-turn transfer seems to rely heavily on how well the supervised contexts match the visited contexts during attack trajectories. I notice that the theory tries to capture this gap. However, the training context is a single-turn harmful request and the deployment context contains the full multi-turn history. I am not clear how stable this correspondence is in practice, or whether the resulting context shift can indeed be summarized by a small transfer slack. The current experiments mainly report downstream ASR and lack analysis of relevant quantities.
2. RLPE seems to rely on policy-ranked high-probability alternatives as proxies for the unsafe mass, without an explicit semantic criterion. This paper describes the likelihood gap as a contrastive selection signal, not a harmfulness label, and that proxy-token suppression depends on the current policy mass and the mixed target. I am therefore unsure why these alternatives should reliably represent the unsafe choices that need to be suppressed, especially when some of them may be benign. Theorem 4.4 connects this mechanism to trajectory risk through proxy coverage and additional tokenwise suppression conditions, but these conditions appear to be assumed instead of being implied by the RLPE objective.
Questions for authors
please see the weaknesses above. In particular:
1. How well do the supervised contexts match the contexts actually visited during multi-turn attacks? Can you report coverage or transfer slack beyond final ASR?
2. How often do RLPE’s policy-ranked proxy tokens actually overlap with unsafe alternatives, and how often do they include benign ones?
3. Theorem 4.4 assumes unsafe proxy tokens are suppressed and the remaining unsafe tokens do not increase. How often do these conditions hold after training?
4. changes position selection, proxy support, and loss normalization at the same time. Can you separate these effects to show what is driving the gain?