acceptodds
Under review as a conference paper at ICLR 2027

Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectories

11 views

Abstract

Large Reasoning Models possess remarkable capabilities for self-correction in general domain; however, they frequently struggle to recover from unsafe reasoning trajectories under adversarial attacks. Existing alignment methods attempt to mitigate this vulnerability by fine-tuning the model on expert data including reflection traces or adversarial prefixes. Crucially, these approaches are often hindered by static training data which inevitably deviate from model's dynamic, on-policy reasoning traces, resulting in model hardly covering its vast generation space and learning to recover from its own failures. To bridge this gap, we propose Self-ReSET, a pure reinforcement learning framework designed to equip LRMs with the intrinsic capacity to recover from their own safety error trajectories, which are subsequently reused as an initial state for reinforcement learning. Extensive experiments across various LRMs and benchmarks demonstrate that Self-ReSET significantly enhances robustness against adversarial attacks especially out-of-distribution (OOD) jailbreak prompts while maintaining general utility, along with efficient data utilization. Further analysis reveals that our method effectively fosters self-recovery patterns, enabling models to better identify and recover from unsafe intermediate error states back to benign paths. Our codes and data are available at https://anonymous.4open.science/r/Self-ReSET-118A.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LRM, safety alignment, recovery, RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_smNEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets a known failure mode in large reasoning models: once a trace turns unsafe, the model rarely returns. Prior RL work trains recovery from externally constructed unsafe prefixes. This paper instead collects prefixes from the model’s own rollouts. Specifically, a streaming guard labels tokens, the trace is cut at the first unsafe token, and the prefix enters a FIFO buffer that is replayed as the starting state for DAPO training.

Critical strengths
  1. The main question of the paper, which unsafe states the mdeol should practice recovering from, is a valuable question following previous work.
  2. Evaluation protocal is well-defined. Training uses Qwen3Guard, and final scoring uses LlamaGuard-3 and GPT4o. This setup avoids reward hacking on the training guard.
  3. Improvements on OOD jaibreaks are significant and consistent across different backbones. H-CoT moves signifcantly from 0, which no baseliens achieve such score.
  4. The paper clearly measures the mechanism it claims and run enough ablation studies to locate the performance improvement.
Critical weaknesses
  1. One central claim is not isolated. The paper claims that continually refreshed on-policy states are better than static ones, but the key control is missing: prefixes self-generated by the initial policy, collected once and never updated. Without this ablation test, “self-generated” and “continually refreshed” remain coupled. In the RECAP reproduction, all three backbones use prefixes generated by DS-Qwen-7B, which is a cross-model setup for two of them.
  2. A simpler explanation might exists. DAPO requires reward variance within a rollout group. Rollouts starting from unsafe prefixes are more likely to produce non-degenerate groups, so the gain may come from denser gradient signal.
  3. The truncation point is inconsistent with the detection rule. A trajectory is flagged only when two consecutive tokens are labeled unsafe (Eq. 2). But truncation uses the earliest unsafe label anywhere in the trace, which need not be that position. An isolated spurious label can thus set the recovery starting state.
  4. The objective shapes exit from unsafe states but not entry. The reward in Eq. 3 depends only on the final response, so entering unsafe reasoning and recovering scores the same as never entering.
Questions for authors
  1. Can you add a control with prefixes self-generated by the initial policy as comparison?
  2. What fraction of rollout groups are non-degenerate under DAPO, with and without replay? Also, what are the total rollout counts and gradient steps for each method?
  3. Is taking t* as the earliest single unsafe label, rather than the earliest position satisfying the consecutive condition, a design choice or just an implementation detail?
  4. What fraction of traces are flagged unsafe by the guard before and after training? During training, what share of updates comes from the buffer versus from fresh prompts?
4 comments
AuthorsOpenReview ↗

Response to Weakness1/Question1 & Weakness2/Question2

Thank you for recognizing the importance of our research question, the clarity of our evaluation protocol, and the consistent improvements across benchmarks and backbones. We also appreciate your positive assessment of our mechanism analyses and ablations. Your questions help us clarify how the components of Self-ReSET work together, and we provide additional controls and training analyses below.

W1 / Q1: Self-Generated Frozen Prefix Control

W1: One central claim is not isolated. The paper claims that continually refreshed on-policy states are better than static ones, but the key control is missing: prefixes self-generated by the initial policy, collected once and never updated. Without this ablation test, “self-generated” and “continually refreshed” remain coupled. In the RECAP reproduction, all three backbones use prefixes generated by DS-Qwen-7B, which is a cross-model setup for two of them.

Q1: Can you add a control with prefixes self-generated by the initial policy as comparison?

Thank you for suggesting this control to separate self-generation from continual refresh. Our additional experiments show that continually refreshing self-generated prefixes improves performance over keeping them fixed, supporting our design choice of updating recovery states as the policy evolves.

To isolate the benefit of continual refresh, we evaluate frozen initial-policy prefixes on DS-Llama-8B and Qwen3-8B. For each model, we collect 750 self-generated prefixes before training and keep them fixed in a buffer. Half of the training samples come from this buffer throughout training, with the same training budget as Self-ReSET.

As shown in Table 1, Self-ReSET outperforms this frozen control on most metrics for DS-Llama-8B and all metrics for Qwen3-8B. Together with the comparison with RECAP on DS-Qwen-7B reported in the paper, these results consistently demonstrate the advantage of continual refresh across all three backbones.

Table 1: Comparison between frozen initial-policy prefixes and continually refreshed Self-ReSET prefixes.

BackboneMethodSR ↑\uparrowHB ↑\uparrowWJ ↑\uparrowSU ↑\uparrowJB-R1 ↑\uparrowFortress ↑\uparrowH-CoT ↑\uparrowXSTest ↑\uparrow
DS-Llama-8BSelf-generated (frozen)98.1695.1990.7091.0095.4577.6010.0092.80
DS-Llama-8BSelf-ReSET98.0098.4094.6097.3098.1087.7037.0098.40
Qwen3-8BSelf-generated (frozen)99.8489.0087.0094.3892.5767.352.0093.60
Qwen3-8BSelf-ReSET100.0098.1095.1098.0097.4077.2021.0096.40

W2 / Q2: Reward Variance and Gradient Updates

W2: A simpler explanation might exists. DAPO requires reward variance within a rollout group. Rollouts starting from unsafe prefixes are more likely to produce non-degenerate groups, so the gain may come from denser gradient signal.

Q2: What fraction of rollout groups are non-degenerate under DAPO, with and without replay? Also, what are the total rollout counts and gradient steps for each method?

Thank you for this observation about reward variance. Denser gradient signals are indeed a key advantage of Self-ReSET: by continually replaying the model's own unsafe intermediate states, the method turns these states into opportunities to learn recovery. Training on them provides both denser gradient signals and informative recovery action tokens, rather than simply increasing the rollout count.

To examine this effect quantitatively, we compare the fraction of non-degenerate groups and the number of gradient updates. Self-ReSET increases the fraction of non-degenerate groups across all three backbones while maintaining a comparable number of updates. Table 2 reports the group statistics with and without replay.

As described in Appendix A.3.2 of the paper, Self-ReSET uses half as many data epochs for rollouts as the other methods. On Qwen3-8B, where Self-ReSET achieves its largest relative improvement, DAPO, RECAP, and Self-ReSET complete 25, 27, and 24 gradient updates, respectively. DAPO and RECAP have already converged at these points. Self-ReSET achieves stronger performance with fewer updates, demonstrating its training efficiency.

Table 2: Fraction of non-degenerate rollout groups with and without replay.

BackboneDAPO (without replay)Self-ReSET (with replay)
DS-Qwen-7B30.65%40.58%
DS-Llama-8B14.75%25.14%
Qwen3-8B11.03%17.97%
AuthorsOpenReview ↗

Response to Weakness3/Question3 & Weakness4/Question4

W3 / Q3: Truncation at the Earliest Unsafe Token

W3: The truncation point is inconsistent with the detection rule. A trajectory is flagged only when two consecutive tokens are labeled unsafe (Eq. 2). But truncation uses the earliest unsafe label anywhere in the trace, which need not be that position. An isolated spurious label can thus set the recovery starting state.

Q3: Is taking t* as the earliest single unsafe label, rather than the earliest position satisfying the consecutive condition, a design choice or just an implementation detail?

Thank you for carefully examining our design. Truncating at the earliest unsafe token is a design choice that provides a natural, earlier starting point for recovery training. Our comparison shows that this choice introduces no obvious additional noise: the two truncation positions yield nearly identical unsafe-completion rates on both backbones.

To compare the two positions, we collect 50 prefixes from each of DS-Llama-8B and DS-Qwen-7B using the earliest single unsafe label and the consecutive-unsafe-label position. We then sample 128 continuations per prefix and measure their unsafe-completion rates. As controls, we use equal-length benign prefixes from the same prompts. Table 3 reports the results.

Table 3: Unsafe-completion rates under different truncation positions and benign controls.

BackboneEarliest unsafe prefixConsecutive-unsafe prefixMatched benign prefix
DS-Llama-8B35.70%36.60%4.69%
DS-Qwen-7B30.47%29.21%4.59%

Both truncation rules yield substantially higher unsafe-completion rates than the benign controls. The close agreement between the two rules confirms that truncating at the earliest unsafe token adds no meaningful noise, supporting its use as a clean and natural recovery boundary.

W4/Q4: Unsafe Entry and Training Composition

W4: The objective shapes exit from unsafe states but not entry. The reward in Eq. 3 depends only on the final response, so entering unsafe reasoning and recovering scores the same as never entering.

Q4: What fraction of traces are flagged unsafe by the guard before and after training? During training, what share of updates comes from the buffer versus from fresh prompts?

To address your question about entry into unsafe reasoning, we examine the unsafe-trajectory rate before and after training. Self-ReSET substantially reduces this rate on both benchmarks and all three backbones, showing that the benefits of recovery training extend to the reasoning process itself.

For this analysis, we use Qwen3Guard-Stream alone to identify unsafe trajectories on WildJailbreak and Fortress. Table 4 reports the fraction of trajectories flagged as unsafe before and after training.

Table 4: Unsafe-trajectory rates before and after Self-ReSET training.

BackboneBenchmarkBase (before training)Self-ReSET (after training)
DS-Llama-8BWildJailbreak91.57%42.97%
DS-Llama-8BFortress79.92%52.71%
DS-Qwen-7BWildJailbreak89.96%53.01%
DS-Qwen-7BFortress76.55%55.22%
Qwen3-8BWildJailbreak51.41%24.90%
Qwen3-8BFortress61.24%44.89%

For your question about training composition, we report the proportions of accepted updates from replay and fresh prompts in Table 5.

Table 5: Proportions of accepted updates from buffer replay and fresh prompts.

BackboneBuffer replayFresh prompts
DS-Llama-8B46.72%53.28%
DS-Qwen-7B57.58%42.42%
Qwen3-8B54.56%45.44%

Thanks for your rebuttal and I will keep my score

Thanks for your response and for your positive assessment of our paper. We appreciate it.

Reviewer_dAw8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an interesting reinforcement learning framework that trains reasoning models to recover from unsafe intermediate reasoning states. The key idea is that a safety guard identifies where the model generated trajectory went and became unsafe, and the corresponding prefix is stored in a replay buffer. The model then samples continuations from these prefixes and is optimized using outcome-based safety rewards. By continually collecting the model’s own failures, the method adapts recovery training to its evolving behavior. Experiments demonstrate improved jailbreak robustness while largely preserving mathematical reasoning performance and benign-request compliance.

Critical strengths
  1. The idea of reusing self-generated unsafe prefixes creates targeted opportunities to do recovery and integrates naturally with existing RL training, which is interesting. Basically, continuously collecting failure states is an interesting way to reduce the mismatch between fixed training prefixes and the errors encountered by an evolving policy.
  2. Experiments are comprehensive and experiments across multiple models and jailbreak benchmarks, together with recovery analyses, over-refusal evaluation, and mathematical reasoning tests, provide evidence of effectiveness of the approach.
  3. Overall, the paper addresses the important challenge of recovering after reasoning has entered an unsafe state, extending safety training beyond initial recognition and refusal of harmful requests.
Critical weaknesses
  1. Resetting to an unsafe prefix creates an opportunity to practice recovery and is an interesting thought, but it does not guarantee that the model can discover a successful safe recovery right? In several hard cases, the model might still be unsafe at some phase or the other if generated in a fully on-policy way leading to sparse grpo rewards, which requires some more analysis on real challenging domains. For example - How frequently do recovery groups contain only unsuccessful continuations, and how does this affect sampling cost and learning?
  2. The replay buffer introduces a mismatch between recovery-training states and states encountered by the current policy. How sensitive is performance to buffer age and capacity, and does refreshing prefixes improve over a frozen collection under matched compute?
  3. The method assumes that guard-flagged prefixes identify meaningful recovery states. Can the authors validate this assumption by sampling multiple continuations from flagged and unflagged prefixes and measuring their unsafe-completion probabilities? This would distinguish prefix-level safety classification from prediction of downstream failure. The approach relies heavily on detecting states from which unsafe outcomes are likely, which implicitly requires anticipating the consequences of continuing the trajectory. The reliability of this detection, particularly for long reasoning trajectories, warrants more detailed ablations and analysis. Note: I am willing to increase my score, based on the above answers as i do believe this work has potential.
Questions for authors

Mentioned in Weakness

7 comments
AuthorsOpenReview ↗

Response to Weakness1

Thank you for describing Self-ReSET as “interesting” and for recognizing its potential and empirical effectiveness. We are greatly encouraged by your assessment. We appreciate your careful questions about the recovery mechanism, which help us explain our design more clearly. Below, we provide additional training analyses and ablations on reward composition, replay freshness, and guard-selected prefixes; these results strongly support our design choices.

W1: Resetting to an unsafe prefix creates an opportunity to practice recovery and is an interesting thought, but it does not guarantee that the model can discover a successful safe recovery right? In several hard cases, the model might still be unsafe at some phase or the other if generated in a fully on-policy way leading to sparse grpo rewards, which requires some more analysis on real challenging domains. For example - How frequently do recovery groups contain only unsuccessful continuations, and how does this affect sampling cost and learning?

Thank you for carefully examining the learning signals available from recovery groups. Our training analysis shows that recovery groups contain a higher proportion of all-failure groups than normal training groups, yet provide denser training signals and higher data efficiency overall. Specifically, they produce more mixed-reward groups with nonzero reward variance, along with high-quality recovery action tokens.

To examine these learning signals, we track the proportions of all-failure, all-success, and mixed-reward groups during Self-ReSET training on DS-Llama-8B. Table 1 reports these proportions for recovery and normal training groups in the first and second halves of training.

Table 1: Reward composition of recovery and normal training groups on DS-Llama-8B.

Training stageGroup sourceAll-failureAll-successMixed reward
First halfRecovery20.61%24.04%55.35%
Normal training1.82%64.51%33.66%
Second halfRecovery9.53%27.93%62.54%
Normal training0.09%95.12%4.79%

As shown in Table 1, recovery groups contain a substantially larger proportion of mixed-reward groups than normal training groups. Therefore, more sampled groups provide nonzero gradient signals, and fewer are discarded because of degenerate rewards. This improves data utilization and makes more effective use of the sampling cost.

The difference becomes more pronounced as training progresses. Among recovery groups, the all-failure proportion decreases, while the mixed-reward proportion increases. In contrast, normal training groups become dominated by all-success groups, with a declining proportion of mixed-reward groups. These trends show that continually refreshed prefixes provide increasingly informative rewards as the model improves. This helps explain the effectiveness of Self-ReSET: continual refresh keeps providing high-quality training signals as the policy evolves.

AuthorsOpenReview ↗

Response to Weakness2

W2: The replay buffer introduces a mismatch between recovery-training states and states encountered by the current policy. How sensitive is performance to buffer age and capacity, and does refreshing prefixes improve over a frozen collection under matched compute?

Thank you for raising this question. Choosing the buffer capacity involves a key trade-off between keeping prefixes close to the current policy and retaining diverse failure states. Our results show that continual refresh is most effective when the buffer preserves both freshness and diversity, supporting C=256C=256 as the default capacity. The frozen-prefix comparison further shows that Self-ReSET consistently improves recovery-related safety while maintaining strong benign compliance.

To examine the balance between freshness and diversity, we measure the mean and 95th-percentile ages of sampled prefixes at different buffer capacities. Smaller buffers provide fresher prefixes; larger buffers retain more failure states but introduce greater policy lag. Table 2 reports these measurements, complementing the capacity ablation in Section 5.4.1 and Table 2 of the paper.

Table 2: Prefix age and policy lag under different replay-buffer capacities.

Buffer capacityMean ageP95 ageSamples with age ≥3\geq 3
C=64C=641.0010%
C=∞C=\infty2.31538.5%
C=256C=2561.1920%

When the buffer capacity equals the batch size, prefixes are used immediately. In contrast, an unbounded buffer introduces substantial policy lag. Our default capacity keeps this lag low while retaining a diverse collection of valuable failure states.

To evaluate the benefit of continual refresh, we compare Self-ReSET with a frozen collection of self-generated trajectories from DS-Llama-8B and Qwen3-8B. For each model, the collected trajectories remain fixed in the buffer, and half of each training batch is sampled from it. We use the same training budget as Self-ReSET and report the safety and over-refusal results in Table 3.

Table 3: Comparison between frozen self-generated prefixes and continually refreshed Self-ReSET prefixes.

BackboneMethodSR ↑\uparrowHB ↑\uparrowWJ ↑\uparrowSU ↑\uparrowJB-R1 ↑\uparrowFortress ↑\uparrowH-CoT ↑\uparrowXSTest ↑\uparrow
DS-Llama-8BSelf-generated (frozen)98.1695.1990.7091.0095.4577.6010.0092.80
DS-Llama-8BSelf-ReSET98.0098.4094.6097.3098.1087.7037.0098.40
Qwen3-8BSelf-generated (frozen)99.8489.0087.0094.3892.5767.352.0093.60
Qwen3-8BSelf-ReSET100.0098.1095.1098.0097.4077.2021.0096.40

These results, together with Table 2 of the paper, demonstrate the benefit of continually refreshing prefixes while maintaining low policy lag and sufficient state diversity. The capacity analysis supports our default C=256C=256: it avoids excessive mixing of states from different policies while preserving the diversity required for effective recovery training.

This finding also complements the training-group analysis in Table 1. As the policy improves, recovery groups provide increasingly informative training samples, and continual refresh keeps the recovery states evolving with the model. By learning from these states, the model obtains useful gradient signals and valuable recovery action tokens from its own unsafe trajectories, enabling iterative improvement throughout training.

AuthorsOpenReview ↗

Response to Weakness3

W3: The method assumes that guard-flagged prefixes identify meaningful recovery states. Can the authors validate this assumption by sampling multiple continuations from flagged and unflagged prefixes and measuring their unsafe-completion probabilities? This would distinguish prefix-level safety classification from prediction of downstream failure. The approach relies heavily on detecting states from which unsafe outcomes are likely, which implicitly requires anticipating the consequences of continuing the trajectory. The reliability of this detection, particularly for long reasoning trajectories, warrants more detailed ablations and analysis.

Thank you for suggesting that we validate selected prefixes through their continuations. This comparison confirms that guard-flagged prefixes identify states with elevated downstream risk from which the model can still recover. These properties make the selected states particularly useful for targeted recovery training.

To make this comparison, we collect 50 training-set prefixes for each of DS-Llama-8B and DS-Qwen-7B using our buffer construction rule. As in the paper, each unsafe reasoning trajectory is truncated at its earliest unsafe token. We pair each flagged prefix with an equal-length, unflagged prefix from the same prompt, with all tokens labeled safe by the guard. We then sample 128 continuations from each prefix and measure the fraction with an unsafe final response. Table 4 reports the averages over the 50 prefixes.

Table 4: Unsafe-completion rates from flagged and matched unflagged prefixes.

BackboneUnsafe-completion rate from flagged prefixesUnsafe-completion rate from unflagged prefixes
DS-Llama-8B35.70%4.69%
DS-Qwen-7B30.47%4.59%

Flagged prefixes lead to substantially more unsafe completions than matched unflagged prefixes, confirming that the guard identifies states with elevated downstream risk. And their unsafe-completion rates remain moderate, leaving meaningful opportunities for successful recovery. Therefore, these prefixes provide both the risk needed for targeted recovery practice and the recoverability needed to generate informative mixed-reward signals.

Reviewer_cqq1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method to automatically recover from reasoning failures by looking at trajectories, identifying failures in reasoning, and storing the failures in a replay buffer. It then uses these samples for further training with RLVR to avoid these failures.

Critical strengths
  1. The paper addresses a timely and important topic, and is generally written well with clear examples and motivation.
  2. The evaluation spans 3 models and benchmarks, showing more or less consistent gains across them.
  3. The algorithm does not require ground truth for fixing the reasoning failures (e.g. it just needs to identify the failed trajectories, and then the final verdict if the response is aligned)
Critical weaknesses
  1. It uses existing components, e.g., DAPO for fine-tuning, judge LLMs for identifying reasoning failures, and then reward generation. The novelty needs to be argued better.
  2. The evaluation mainly spans small models, so it is difficult to understand if it has the desired effect in larger models.
  3. The replay buffer stores only a limited number of unsafe reasoning prefixes. I am concerned that the model might become too focused on these specific failures and perform poorly on other reasoning failures that are not captured in the buffer.
  4. The paper does not address the issue of adaptive attacks (e.g. Autodan turbo), which should be addressed in an LLM safety paper. If it uses prefixes to identify trajectories, it may not work for adaptive adversaries.
  5. The reward is based on final prompt response only, so it is unclear if the reasoning is actually getting improved through DAPO.
Questions for authors
  1. Section 5.2 claims that models with stronger reasoning capabilities benefit more from self-recovery training. Could you evaluate at least one larger model (e.g., 14B or 32B) on a subset of benchmarks to support this claim?
  2. RECAP is closely related. The main difference seems to be on-policy, continuously refreshed prefixes versus static prefilled. Could you isolate this by running your pipeline with prefixes collected once at the start of training and kept fixed, so the gain from on-policy collection is measured directly?
  3. Is it possible to concretely show that the final reasoning is getting better, not just that the final response is more aligned? e.g. how many safe final responses still contain harmful content in the reasoning.
  4. If possible, use a small subset and run an adaptive attack with (1) your defense and (2) the strongest baseline.
4 comments
AuthorsOpenReview ↗

Response to Weakness1 & Weakness2 & Question1

Thank you for recognizing the timeliness and importance of our research question and the consistent improvements across three models. We also appreciate your recognition that Self-ReSET learns without ground-truth recovery trajectories. Your questions give us an opportunity to clarify our contribution and strengthen its empirical support, and we address them below with additional experiments.

W1: Further Novelty Clarification

W1: It uses existing components, e.g., DAPO for fine-tuning, judge LLMs for identifying reasoning failures, and then reward generation. The novelty needs to be argued better.

We would like to further clarify the novelty of Self-ReSET. The key idea is to construct recovery states from the current policy's own unsafe intermediate trajectories and continually refresh them as the policy evolves. Training from these states teaches the model to recover as its reasoning approaches an unsafe trajectory, improving safety alignment through RL. The comparison with frozen self-generated prefixes in Q2 further supports the novelty and effectiveness of this design: continual refresh provides benefits beyond self-generation alone.

Prior recovery methods rely on static, externally constructed trajectories or fixed unsafe prefills. These states cover a limited portion of the failure space and can diverge from the errors produced by the current policy. Self-ReSET addresses this mismatch by allowing recovery states to co-evolve with the policy, improving both reasoning safety and final-response alignment. To our knowledge, it is the first method to organize recovery training around continually refreshed, on-policy unsafe prefixes collected from the policy itself. This design addresses a question largely overlooked by prior work: which unsafe states should the model practice recovering from?

Within this recovery-training framework, DAPO provides the optimization method, while judge LLMs provide safety labels. These components are also used in other safety-alignment methods: RECAP uses DAPO for optimization [1], and LLM-based guards are widely used to provide safety labels and learning signals [2–4]. Self-ReSET builds on these tools to train on recovery states that are constructed and continually refreshed from the policy itself. This training approach improves the model's final safety performance and is the core contribution validated by our experiments.

W2 / Q1: Effectiveness on Larger Models

W2: The evaluation mainly spans small models, so it is difficult to understand if it has the desired effect in larger models.

Q1: Section 5.2 claims that models with stronger reasoning capabilities benefit more from self-recovery training. Could you evaluate at least one larger model (e.g., 14B or 32B) on a subset of benchmarks to support this claim?

Thank you for suggesting an evaluation at a larger scale. Our additional experiments on DS-Qwen-14B show that Self-ReSET retains its advantage: it achieves the best performance on the reported OOD jailbreak benchmarks and substantially improves recovery-related safety. At the same time, benign instruction following and general capabilities remain close to those of the base model.

Compared with DS-Qwen-7B in the main table, RECAP provides more limited gains over DAPO on DS-Qwen-14B, while Self-ReSET continues to yield substantial improvements. This is consistent with Section 5.2: continually refreshing recovery states allows recovery training to co-evolve with the policy, helping stronger models effectively exploit self-recovery signals and benefit more than they do from recovery training with static prefixes.

Table 1 reports the safety results and a representative set of general-capability results.

Table 1: Safety and general-capability evaluation on DS-Qwen-14B.

MethodSRHBWJSUJB-R1FortressH-CoTXSTestM500MMLUHE+
Base61.9040.0058.6081.1361.4247.202.5091.6090.9974.5383.54
STAR-199.5297.1381.5081.7593.3772.550.0088.4090.7374.6983.54
SafeChain79.9554.6967.0077.2571.5755.354.5098.4090.1474.5785.37
DAPO99.3698.2588.8088.8894.4170.200.5096.0091.2174.4881.71
RECAP98.9693.6386.2089.8892.3366.401.0096.4091.6974.5181.10
Self-ReSET99.9296.8196.4097.8899.0484.3019.0094.4091.5874.5485.37
AuthorsOpenReview ↗

Response to Question2 & Weakness3

Q2: Continually Refreshed versus Frozen Self-Generated Prefixes

Q2: RECAP is closely related. The main difference seems to be on-policy, continuously refreshed prefixes versus static prefilled. Could you isolate this by running your pipeline with prefixes collected once at the start of training and kept fixed, so the gain from on-policy collection is measured directly?

Thank you for suggesting this control to distinguish continual refresh from self-generation. Our comparison shows that continually refreshing self-generated prefixes yields stronger recovery-related safety than keeping them fixed, isolating the benefit of updating recovery states as the policy evolves.

Specifically, we collect 750 unsafe prefixes from each model before training using our collection rule and keep them fixed throughout training. We then use the same pipeline, drawing half of each training batch from this frozen buffer.

As shown in Table 2, Self-ReSET outperforms the frozen control on most metrics for DS-Llama-8B and all metrics for Qwen3-8B. The gains are especially clear on WJ, SU, JB-R1, Fortress, and H-CoT. Since both settings use self-generated prefixes, these results isolate the additional benefit of continual refresh.

Table 2: Comparison with a frozen buffer of initial-policy self-generated prefixes.

BackboneMethodSR ↑\uparrowHB ↑\uparrowWJ ↑\uparrowSU ↑\uparrowJB-R1 ↑\uparrowFortress ↑\uparrowH-CoT ↑\uparrowXSTest ↑\uparrow
DS-Llama-8BSelf-generated (frozen)98.1695.1990.7091.0095.4577.6010.0092.80
DS-Llama-8BSelf-ReSET98.0098.4094.6097.3098.1087.7037.0098.40
Qwen3-8BSelf-generated (frozen)99.8489.0087.0094.3892.5767.352.0093.60
Qwen3-8BSelf-ReSET100.0098.1095.1098.0097.4077.2021.0096.40

W3: Generalization beyond the Replay Buffer

W3: The replay buffer stores only a limited number of unsafe reasoning prefixes. I am concerned that the model might become too focused on these specific failures and perform poorly on other reasoning failures that are not captured in the buffer.

Thank you for raising this question. Our capacity ablation in Section 5.4.1 shows that an unbounded buffer retains samples with excessive policy lag and weakens performance. Therefore, expanding the buffer without a capacity limit to collect more trajectories can harm training.

We also analyze whether the recovery ability learned by Self-ReSET generalizes beyond the buffer. The results show that it extends to unsafe reasoning patterns missed by the buffer construction guard, rather than being restricted to the specific failures stored during training.

To examine this generalization, we collect model trajectories on WildJailbreak and evaluate reasoning safety with Qwen3Guard-Stream and GPT-4o. We select trajectories whose reasoning is judged unsafe by GPT-4o but safe by Qwen3Guard-Stream. This subset represents unsafe reasoning patterns missed by the buffer construction guard. We then measure the fraction of these trajectories that end with a safe final response, following the paper's definition of recovery rate.

As shown in Table 3, Self-ReSET achieves the highest recovery rate on all three backbones.

Table 3: Recovery rates on unsafe reasoning trajectories missed by Qwen3Guard-Stream.

BackboneBaseDAPORECAPSelf-ReSET
DS-Llama-8B8.8%10.0%15.4%45.0%
DS-Qwen-7B2.7%18.2%14.0%38.7%
Qwen3-8B16.1%25.7%22.7%71.8%
AuthorsOpenReview ↗

Response to Weakness4 & Question4 & Weakness5 & Question3

W4 / Q4: Robustness to Adaptive Attacks

W4: The paper does not address the issue of adaptive attacks (e.g. Autodan turbo), which should be addressed in an LLM safety paper. If it uses prefixes to identify trajectories, it may not work for adaptive adversaries.

Q4: If possible, use a small subset and run an adaptive attack with (1) your defense and (2) the strongest baseline.

Following your suggestion, we additionally evaluate robustness to adaptive attacks. Self-ReSET achieves the lowest attack success rate at each query budget on all three backbones, showing that recovery training also strengthens the model's ability to resist these attacks.

For this evaluation, we use AutoDAN-Turbo with the same 100 StrongREJECT requests and a frozen public strategy library for all three backbones. For each target model, the attacker retrieves strategies based on that model's trajectories from the previous round and uses them to construct the next attack. We use Llama-3.1-8B-Instruct to generate attacks and Qwen3-32B to score the search. Each request receives 50 target queries; Table 4 reports ASR@10, ASR@25, and ASR@50.

Table 4: AutoDAN-Turbo attack success rates under different query budgets.

BackboneMethodASR@10 ↓\downarrowASR@25 ↓\downarrowASR@50 ↓\downarrow
DS-Llama-8BBase83%97%99%
STAR-137%72%91%
SafeChain73%91%97%
DAPO38%67%87%
RECAP27%48%65%
Self-ReSET19%33%50%
DS-Qwen-7BBase87%97%98%
STAR-156%85%95%
SafeChain77%91%98%
DAPO49%75%93%
RECAP53%81%95%
Self-ReSET38%60%78%
Qwen3-8BBase79%92%98%
STAR-163%87%97%
SafeChain68%93%98%
DAPO71%93%99%
RECAP67%94%99%
Self-ReSET44%83%94%

W5 / Q3: Reasoning Safety beyond Final Responses

W5: The reward is based on final prompt response only, so it is unclear if the reasoning is actually getting improved through DAPO.

Q3: Is it possible to concretely show that the final reasoning is getting better, not just that the final response is more aligned? e.g. how many safe final responses still contain harmful content in the reasoning.

We appreciate your suggestion to examine reasoning safety directly, including reasoning that accompanies a safe final response. This analysis is necessary under our reward setting, which evaluates only the final response.

Our results below show that Self-ReSET reduces unsafe reasoning overall while maintaining consistency between reasoning and final responses. Its improvement in final-response safety does not come at the expense of reasoning safety.

To assess reasoning safety independently of the training guard, we collect WildJailbreak trajectories from the evaluated checkpoints. We use GPT-4o to evaluate reasoning safety and LlamaGuard3-8B to evaluate final-response safety. Table 5 reports the overall unsafe-reasoning rate and the unsafe-reasoning rate among safe final responses. Self-ReSET achieves the lowest overall unsafe-reasoning rate on all three backbones. Among safe final responses, it also achieves the second-lowest unsafe-reasoning rate on each backbone, consistently below Base and RECAP.

Table 5: Reasoning safety and consistency between reasoning and final responses on WildJailbreak.

BackboneMethodUnsafe reasoningUnsafe reasoning among safe final responses
DS-Llama-8BBase71.60%63.03%
DAPO19.68%12.90%
RECAP18.80%16.82%
Self-ReSET15.60%13.10%
DS-Qwen-7BBase74.40%63.30%
DAPO30.80%17.65%
RECAP43.60%36.97%
Self-ReSET28.40%26.07%
Qwen3-8BBase60.80%48.51%
DAPO30.00%27.07%
RECAP37.60%32.94%
Self-ReSET28.40%28.44%

References

[1] ShengYun Peng et al. Large Reasoning Models Learn Better Alignment from Flawed Thinking. arXiv:2510.00938. https://arxiv.org/abs/2510.00938

[2] Yi Zhang et al. AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning. arXiv:2507.14987. https://arxiv.org/abs/2507.14987

[3] Taehan Kim et al. InvThink: Premortem Reasoning for Safer Language Models. arXiv:2510.01569. https://arxiv.org/abs/2510.01569

[4] Seanie Lee et al. ThinkSafe: Self-Generated Safety Alignment for Reasoning Models. arXiv:2601.23143. https://arxiv.org/abs/2601.23143

AuthorsOpenReview ↗

Follow-up on the rebuttal to reviewer cqq1

Dear reviewer cqq1,

We hope this finds you well. As the discussion period is approaching its end in fewer than two days, we would like to respectfully confirm whether we have sufficiently addressed your concerns.

In our rebuttal, we clarified the novelty and contribution of continual refresh, with additional experiments showing its advantage over frozen prefixes and its effectiveness on a larger model and against adaptive attacks. Our analyses also show that recovery generalizes to unsafe reasoning patterns missed by the buffer and that Self-ReSET reduces unsafe reasoning without substantially compromising consistency with final responses.

If there are any remaining questions requiring further clarification, please don't hesitate to let us know. Your insights are valuable to us, and we are eager to address any issues to improve our work.

Thanks for your time and effort in reviewing our paper.

Best,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response for summarizing our rebuttal

Dear Area Chair and Reviewers,

Thanks for your constructive feedback. We appreciate the reviewers' recognition of the important research question and consistent improvements across backbones (cqq1{\color{blue}{cqq1}} and smNE{\color{green}{smNE}}). We also thank Reviewer dAw8{\color{red}{dAw8}} for describing the approach as interesting, recognizing its potential, and highlighting the value of adapting recovery training to the evolving policy.

Self-ReSET trains reasoning models to recover from their own unsafe intermediate states by continually refreshing recovery prefixes as the policy evolves.

During the discussion, we have provided point-by-point responses and additional experiments to address the reviewers' concerns. Reviewer smNE{\color{green}{smNE}} has maintained their positive recommendation following our rebuttal. Reviewer dAw8{\color{red}{dAw8}} found the additional experiments helpful and recognized the frozen-buffer comparison as good evidence for continual refresh. We have since answered both follow-up questions on sampling overhead and prefix collection. Reviewer cqq1{\color{blue}{cqq1}} has not yet responded to our rebuttal, which provides additional experiments and point-by-point responses to their concerns.

Below, we summarize how the additional experiments and analyses address the main concerns:

  1. Continual refresh improves over frozen self-generated prefixes. (cqq1{\color{blue}{cqq1}} W1/Q2; dAw8{\color{red}{dAw8}} W2; smNE{\color{green}{smNE}} W1/Q1) We added the frozen initial-policy prefix control requested by all three reviewers, using the same training pipeline and budget on two backbones. Self-ReSET improves H-CoT safety from 10% to 37% on DS-Llama-8B and 2% to 21% on Qwen3-8B, with additional gains on multiple jailbreak benchmarks. These results directly support the benefit of updating recovery states as the policy evolves.

  2. The benefits extend to a 14B model. (cqq1{\color{blue}{cqq1}} W2/Q1) We added DS-Qwen-14B experiments, where Self-ReSET achieves the best performance on the reported OOD jailbreak benchmarks while preserving the evaluated general capabilities. These results extend the evidence beyond the original 7B–8B models.

  3. Self-ReSET improves robustness under adaptive attacks. (cqq1{\color{blue}{cqq1}} W4/Q4) We added AutoDAN-Turbo evaluations on all three original backbones. Self-ReSET achieves the lowest attack success rate among the evaluated methods at all reported query budgets, extending the robustness evidence beyond fixed jailbreak benchmarks.

  4. Mechanism analyses support prefix selection and sampling efficiency. (dAw8{\color{red}{dAw8}} W1–W3 and follow-up; smNE{\color{green}{smNE}} W2/Q2 and W3/Q3) Continuation tests validate that selected prefixes identify states with elevated downstream risk and opportunities for successful recovery. Compared with DAPO, Self-ReSET produces a higher proportion of mixed-reward groups. Within Self-ReSET training on DS-Llama-8B, recovery groups also incur lower generation-token costs per accepted mixed-reward group than fresh-prompt groups, including the cost of discarded samples. We also quantify policy lag across buffer capacities, clarifying the balance between replay freshness and diversity.

  5. Safety gains extend to reasoning and failures missed by the training guard. (cqq1{\color{blue}{cqq1}} W3/W5/Q3; smNE{\color{green}{smNE}} W4/Q4) GPT-4o evaluation, independent of the training guard, shows that Self-ReSET achieves the lowest overall unsafe-reasoning rate among the evaluated methods on all three backbones. It also achieves the highest recovery rate on unsafe reasoning missed by the training guard, supporting generalization beyond the failures selected for replay.

Together, these additional controls and evaluations strengthen the empirical support for Self-ReSET's contribution, robustness, and recovery mechanism. We have made every effort to address the reviewers' concerns, including the follow-up questions raised during the discussion. Thank you again to the Area Chair and reviewers for your time and consideration.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.