- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper considers the problem of semantic representation decay in safety training for LLM. While most existing papers apply training on offline benign and harmful prompt directly, the paper focus on the weaknesses of these approaches, which is semantic representation decay. The harmful prompt becomes unseperatable with linear probe as completion got generated. For solving this issue, the authors propose a two stages approach which in the first stage learns an invariant representation of the harmful intent, and pining on the intent during GRPO training in the second stage. Through experiment results, the authors demonstrated the approach with four different models and eight jailbreak attacks on AdvBench.
Critical strengths
- TSC-GRPO addresses the issue with compliant trajectory through GRPO with rollouts from compromised context directly. This directly brings robustness against these kinds of attacks.
- The experiments setup considers multiple model families and sizes with 8 jailbreak techniques. The setup is comprehensive and empirically enough to support some statements in the paper.
Critical weaknesses
- In stage 1, the learned invariant representation never learns about harmfulness, but rather than learning query identity. z_anchor is an embedding of similar queries across harmful continuation context, not really a harmful intent. This means one can still hack the reward with any drift away from the query's embedding. The robustness against unseen attacks becomes unclear.
- The advantage over a standard GRPO with harm-classifier reward is not clear. Since the approach doesn't learn identifying the harmfulness, training a classifier on the same dataset with harmful or benign as labels could provide even more discriminatory information.
- The learned invariant representation might miss-anchored a benign prompt which starts with a harmful-style prompt. Considering a multi-turn conversation starts with a harmful prompt, but gradually move back to benign. With the proposed approach, the model will not be able to change its intent representation gradually, which leads to over-refusal.
Questions for authors
1.HOw does TSC-GRPO compare to GRPO from the same forced contexts using a harmful classifier as reward? 2. Are the +- values standard deviation over multiple runs? which judge determines ASR? 3. If a model was used as the judge for ASR, a further experiments on different judges would make the result more reliable.
5 comments
Response to Weakness 2 & Question 1
We thank the reviewer for suggesting a comparison with harmfulness classifier rewards. The key distinction lies in what the reward evaluates and how it guides optimization. A conventional response-level harmfulness classifier evaluates the safety of the final output. In contrast, TSC-GRPO focuses on safety recovery after generation has entered a high-risk state. Two trajectories may receive the same final safety label but differ in when they stop harmful continuation. Our cumulative trajectory reward captures this temporal difference by measuring the similarity between intermediate representations and the original harmful-semantic anchor. This encourages the policy to interrupt harmful continuation earlier.
This design builds on the context-invariant representation learned in Stage 1. By aligning different contextual views of the same query, Stage 1 provides a stable semantic reference for evaluating intermediate generation states. Combining semantic anchoring with cumulative trajectory feedback enables TSC-GRPO to target the recovery process underlying Shallow Safety Alignment.
The existing ablation results further support the contribution of this reward. As shown in Figure 2(b), removing the trajectory reward () substantially increases ASR, while keeping GRPO training and exposure to high-risk contexts unchanged. This indicates that the improvement is not solely due to GRPO or adversarial context exposure. While this result supports the effectiveness of our trajectory reward, a direct comparison with harmfulness classifier rewards under identical training conditions remains necessary to establish their relative performance. We will clarify this distinction and the scope of our empirical evidence in the revised manuscript.
Response to Weakness 3
We thank the reviewer for raising this important concern about over-refusal and changes in user intent. Intent Pinning encourages semantic consistency under intent-preserving contextual perturbations. As defined in Section 4.3, , where denotes request semantics and represents contextual factors. When user intent changes from harmful to benign, the semantic content also changes and is not subject to the same invariance constraint.
Our training design also considers benign requests. Stage 1 includes both harmful and benign queries, while Stage 2 retains standard on-policy sampling for benign queries. The semantic anchor provides trajectory-level training feedback for a given harmful request, while the trained policy responds to the current conversation context during inference.
To directly examine over-refusal, we compare Qwen2.5-7B-Instruct before and after TSC-GRPO. We evaluate the 250 safe prompts from XSTest and 30 paired multi-turn conversations in which user intent gradually shifts from harmful to benign. Each multi-turn conversation is paired with the same final benign request presented without the preceding harmful context. Both models use identical evaluation settings. The over-refusal rates are reported below.
Table R4: Over-refusal analysis on benign requests and intent transitions.
| Evaluation Setting | Pre Over-refusal (%) | Post Over-refusal (%) |
|---|---|---|
| XSTest Safe (250) | 8.8 | 8.4 |
| Benign-only (30) | 6.7 | 6.7 |
| Harmful → Benign (30) | 13.3 | 13.3 |
As shown in Table R4, the over-refusal rate slightly decreases from 8.8% to 8.4% on XSTest Safe. In the paired multi-turn evaluation, the rates remain unchanged after TSC-GRPO under both benign-only (6.7%) and harmful-to-benign (13.3%) settings. Although preceding harmful contexts increase over-refusal, TSC-GRPO does not further amplify this effect in our evaluation. These results provide preliminary evidence that TSC-GRPO preserves the ability to respond appropriately to benign requests, including those following changes in user intent.
Response to Weakness 1
We thank the reviewer for the insightful comments on the learned representation and the reliability of the trajectory reward. Stage 1 encourages representation consistency across contextual views of the same query while preserving distinctions among queries. The independent validation in Appendix C provides evidence that the learned representations retain harm-relevant information. Under a strict query-level split, the probe achieves 94.24% harmful/benign classification accuracy on unseen queries and 92.6% accuracy under compliant-prefix perturbations. These results support the preservation of harm-relevant information beyond individual training queries.
In Stage 2, the query-specific semantic anchor provides a stable reference for evaluating the evolving generation trajectory. The cumulative reward penalizes sustained above-threshold similarity to the harmful-semantic anchor, encouraging earlier recovery from harmful continuations.
To further examine reward reliability and potential semantic drift, we conduct a paired analysis on 50 held-out AdvBench queries using the frozen Stage-1 scorer. For each query, we compare three types of continuations under the same forced context and semantic anchor: harmful continuation, safe refusal, and off-topic unsafe continuation. We use independent safety annotations to verify the continuation categories and control response lengths when comparing cumulative rewards.
Table R3: Pairwise reward preference on held-out AdvBench queries.
The reward favors safe refusal over harmful continuation in 86.0% of paired cases and over off-topic unsafe continuation in 80.0%. These results show that the trajectory reward generally favors safety recovery over both sustained harmful generation and unsafe semantic drift in the evaluated cases.
Together with the held-out representation validation in Appendix C and the trajectory-reward ablation in Figure 2(b), this analysis provides complementary evidence for the effectiveness of the proposed semantic anchoring mechanism.