- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper targets a known failure mode in large reasoning models: once a trace turns unsafe, the model rarely returns. Prior RL work trains recovery from externally constructed unsafe prefixes. This paper instead collects prefixes from the model’s own rollouts. Specifically, a streaming guard labels tokens, the trace is cut at the first unsafe token, and the prefix enters a FIFO buffer that is replayed as the starting state for DAPO training.
Critical strengths
- The main question of the paper, which unsafe states the mdeol should practice recovering from, is a valuable question following previous work.
- Evaluation protocal is well-defined. Training uses Qwen3Guard, and final scoring uses LlamaGuard-3 and GPT4o. This setup avoids reward hacking on the training guard.
- Improvements on OOD jaibreaks are significant and consistent across different backbones. H-CoT moves signifcantly from 0, which no baseliens achieve such score.
- The paper clearly measures the mechanism it claims and run enough ablation studies to locate the performance improvement.
Critical weaknesses
- One central claim is not isolated. The paper claims that continually refreshed on-policy states are better than static ones, but the key control is missing: prefixes self-generated by the initial policy, collected once and never updated. Without this ablation test, “self-generated” and “continually refreshed” remain coupled. In the RECAP reproduction, all three backbones use prefixes generated by DS-Qwen-7B, which is a cross-model setup for two of them.
- A simpler explanation might exists. DAPO requires reward variance within a rollout group. Rollouts starting from unsafe prefixes are more likely to produce non-degenerate groups, so the gain may come from denser gradient signal.
- The truncation point is inconsistent with the detection rule. A trajectory is flagged only when two consecutive tokens are labeled unsafe (Eq. 2). But truncation uses the earliest unsafe label anywhere in the trace, which need not be that position. An isolated spurious label can thus set the recovery starting state.
- The objective shapes exit from unsafe states but not entry. The reward in Eq. 3 depends only on the final response, so entering unsafe reasoning and recovering scores the same as never entering.
Questions for authors
- Can you add a control with prefixes self-generated by the initial policy as comparison?
- What fraction of rollout groups are non-degenerate under DAPO, with and without replay? Also, what are the total rollout counts and gradient steps for each method?
- Is taking t* as the earliest single unsafe label, rather than the earliest position satisfying the consecutive condition, a design choice or just an implementation detail?
- What fraction of traces are flagged unsafe by the guard before and after training? During training, what share of updates comes from the buffer versus from fresh prompts?
4 comments
Response to Weakness3/Question3 & Weakness4/Question4
W3 / Q3: Truncation at the Earliest Unsafe Token
W3: The truncation point is inconsistent with the detection rule. A trajectory is flagged only when two consecutive tokens are labeled unsafe (Eq. 2). But truncation uses the earliest unsafe label anywhere in the trace, which need not be that position. An isolated spurious label can thus set the recovery starting state.
Q3: Is taking t* as the earliest single unsafe label, rather than the earliest position satisfying the consecutive condition, a design choice or just an implementation detail?
Thank you for carefully examining our design. Truncating at the earliest unsafe token is a design choice that provides a natural, earlier starting point for recovery training. Our comparison shows that this choice introduces no obvious additional noise: the two truncation positions yield nearly identical unsafe-completion rates on both backbones.
To compare the two positions, we collect 50 prefixes from each of DS-Llama-8B and DS-Qwen-7B using the earliest single unsafe label and the consecutive-unsafe-label position. We then sample 128 continuations per prefix and measure their unsafe-completion rates. As controls, we use equal-length benign prefixes from the same prompts. Table 3 reports the results.
Table 3: Unsafe-completion rates under different truncation positions and benign controls.
| Backbone | Earliest unsafe prefix | Consecutive-unsafe prefix | Matched benign prefix |
|---|---|---|---|
| DS-Llama-8B | 35.70% | 36.60% | 4.69% |
| DS-Qwen-7B | 30.47% | 29.21% | 4.59% |
Both truncation rules yield substantially higher unsafe-completion rates than the benign controls. The close agreement between the two rules confirms that truncating at the earliest unsafe token adds no meaningful noise, supporting its use as a clean and natural recovery boundary.
W4/Q4: Unsafe Entry and Training Composition
W4: The objective shapes exit from unsafe states but not entry. The reward in Eq. 3 depends only on the final response, so entering unsafe reasoning and recovering scores the same as never entering.
Q4: What fraction of traces are flagged unsafe by the guard before and after training? During training, what share of updates comes from the buffer versus from fresh prompts?
To address your question about entry into unsafe reasoning, we examine the unsafe-trajectory rate before and after training. Self-ReSET substantially reduces this rate on both benchmarks and all three backbones, showing that the benefits of recovery training extend to the reasoning process itself.
For this analysis, we use Qwen3Guard-Stream alone to identify unsafe trajectories on WildJailbreak and Fortress. Table 4 reports the fraction of trajectories flagged as unsafe before and after training.
Table 4: Unsafe-trajectory rates before and after Self-ReSET training.
| Backbone | Benchmark | Base (before training) | Self-ReSET (after training) |
|---|---|---|---|
| DS-Llama-8B | WildJailbreak | 91.57% | 42.97% |
| DS-Llama-8B | Fortress | 79.92% | 52.71% |
| DS-Qwen-7B | WildJailbreak | 89.96% | 53.01% |
| DS-Qwen-7B | Fortress | 76.55% | 55.22% |
| Qwen3-8B | WildJailbreak | 51.41% | 24.90% |
| Qwen3-8B | Fortress | 61.24% | 44.89% |
For your question about training composition, we report the proportions of accepted updates from replay and fresh prompts in Table 5.
Table 5: Proportions of accepted updates from buffer replay and fresh prompts.
| Backbone | Buffer replay | Fresh prompts |
|---|---|---|
| DS-Llama-8B | 46.72% | 53.28% |
| DS-Qwen-7B | 57.58% | 42.42% |
| Qwen3-8B | 54.56% | 45.44% |
Thanks for your rebuttal and I will keep my score
Thanks
Thanks for your response and for your positive assessment of our paper. We appreciate it.
Response to Weakness1/Question1 & Weakness2/Question2
Thank you for recognizing the importance of our research question, the clarity of our evaluation protocol, and the consistent improvements across benchmarks and backbones. We also appreciate your positive assessment of our mechanism analyses and ablations. Your questions help us clarify how the components of Self-ReSET work together, and we provide additional controls and training analyses below.
W1 / Q1: Self-Generated Frozen Prefix Control
Thank you for suggesting this control to separate self-generation from continual refresh. Our additional experiments show that continually refreshing self-generated prefixes improves performance over keeping them fixed, supporting our design choice of updating recovery states as the policy evolves.
To isolate the benefit of continual refresh, we evaluate frozen initial-policy prefixes on DS-Llama-8B and Qwen3-8B. For each model, we collect 750 self-generated prefixes before training and keep them fixed in a buffer. Half of the training samples come from this buffer throughout training, with the same training budget as Self-ReSET.
As shown in Table 1, Self-ReSET outperforms this frozen control on most metrics for DS-Llama-8B and all metrics for Qwen3-8B. Together with the comparison with RECAP on DS-Qwen-7B reported in the paper, these results consistently demonstrate the advantage of continual refresh across all three backbones.
Table 1: Comparison between frozen initial-policy prefixes and continually refreshed Self-ReSET prefixes.
W2 / Q2: Reward Variance and Gradient Updates
Thank you for this observation about reward variance. Denser gradient signals are indeed a key advantage of Self-ReSET: by continually replaying the model's own unsafe intermediate states, the method turns these states into opportunities to learn recovery. Training on them provides both denser gradient signals and informative recovery action tokens, rather than simply increasing the rollout count.
To examine this effect quantitatively, we compare the fraction of non-degenerate groups and the number of gradient updates. Self-ReSET increases the fraction of non-degenerate groups across all three backbones while maintaining a comparable number of updates. Table 2 reports the group statistics with and without replay.
As described in Appendix A.3.2 of the paper, Self-ReSET uses half as many data epochs for rollouts as the other methods. On Qwen3-8B, where Self-ReSET achieves its largest relative improvement, DAPO, RECAP, and Self-ReSET complete 25, 27, and 24 gradient updates, respectively. DAPO and RECAP have already converged at these points. Self-ReSET achieves stronger performance with fewer updates, demonstrating its training efficiency.
Table 2: Fraction of non-degenerate rollout groups with and without replay.