- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes DACPO for recovering from perturbed reasoning states without an explicit correction cue. It combines supervised correction training with recovery-focused GRPO, using difficult-state selection, failure replay, and adaptive harmful/benign batch mixing. Experiments across four backbones show improvements in both direct and injected safety evaluations, with additional analyses of cue dependence, over-refusal, mathematical accuracy, and individual components.
Critical strengths
-
Clear problem formulation. The paper distinguishes initial safety from recovery after reasoning drift and evaluates cue dependence using matched perturbed states. This provides a useful perspective on safety beyond responses to intact prompts.
-
Consistent empirical improvements. DACPO achieves the lowest or tied-lowest direct and injected ASR across the evaluated backbone–benchmark combinations.
Critical weaknesses
-
The motivation for prioritizing recovery over prevention is insufficiently justified. While DACPO improves both direct and injected ASR, the paper does not establish why training models to recover from unsafe reasoning is preferable to strengthening their ability to avoid it. Injected evaluation forces models into perturbed states regardless of how effectively they prevent such states. A budget-matched comparison with training that aims to avoid unsafe reasoning in the first place would clarify the additional value of recovery-focused training.
-
The adaptive machinery is not sufficiently justified over simple recovery RL. The component ablations are useful, but do not establish an advantage over budget-matched recovery GRPO with fixed drift sampling and a tuned fixed harmful/benign mixture. This comparison would clarify whether the added complexity and probing cost are necessary.
Questions for authors
Q1. Under matched initialization, data, and training budgets, how does DACPO compare with safety training focused on preventing unsafe reasoning rather than recovering from it?
Q2. How does DACPO compare with simple recovery GRPO using fixed sampling and a tuned fixed harmful/benign mixture? What mixture replaces the controller in its ablation?
4 comments
Dear Reviewer Tet8,
Thank you again for your thoughtful review. As the discussion period is nearing its end, we wanted to follow up on our response. If you have a chance to take a look, we would appreciate hearing whether our clarifications address your concerns. We would be happy to answer any remaining questions. Thank you for your time and consideration.
We sincerely look forward to your feedback.
Thank you for your response. The additional experiments and clarification well addressed my concerns and questions. I have raised my score accordingly.
Thank you for taking the time to review our response and reconsider your score. We’re glad the additional experiments and clarifications addressed your concerns. Your feedback helped us improve the paper, and we appreciate your time throughout the review process.
Response to Reviewer
Thank you for asking whether recovery training adds value beyond prevention and how DACPO compares with fixed-sampling recovery GRPO.
Q1. Comparison with prevention-focused training
We compared DACPO with SAFEPATH and ThinkSafe adaptations on DeepSeek-R1-Distill-Qwen-7B. SafeChain, from the submitted Table 1, provides an additional baseline under the same evaluation protocol. DACPO had lower ASR and I-ASR than all three baselines on each benchmark. The unweighted mean ASR and I-ASR were 3.38% and 6.72% for DACPO, 30.30% and 36.29% for SAFEPATH, and 25.48% and 31.36% for ThinkSafe. Against ThinkSafe, the strongest baseline on both means, the gap was already 22.10 percentage points on intact prompts.
On harmful prompts, SAFEPATH supervises a short primer at the start of reasoning. ThinkSafe fine-tunes on full trajectories that pass a safety filter after refusal steering. DACPO probes the current policy to find difficult perturbed states and trains the model to continue safely from them. This difference in training exposure offers one explanation for the performance gap on shared perturbed prefixes. Direct ASR reflects both prevention and recovery during generation. The shared-prefix result supports training for recovery alongside prevention because it measures safety after all models have reached the same perturbed state. Direct safety and recovery from harmful drift on DeepSeek-R1-Distill-Qwen-7B. Both ASR and I-ASR use avg@4. The first three benchmarks evaluate harmful prompts. SafeUnlearning and WJ-AdvHarm evaluate jailbreak prompts.
Q2. Comparison with fixed-sampling recovery GRPO
Table 5 controller ablation. We replaced adaptive batch mixing with a fixed 8:2 ratio of harmful to benign state groups throughout Stage II and retained adaptive drift-state selection and failure replay.
Fixed-sampling recovery GRPO. We drew states from a frozen pool built from the initial drift archive and kept the correction-SFT checkpoint, source prompt pool, recovery rewards, and GRPO settings shared with DACPO. Of the fixed-sampling settings reported below, 6:4 was closest to DACPO on both XSTest refusal rates (16.10% ORR and 21.55% I-ORR versus 15.40% and 21.80%). DACPO achieved 4.17 percentage points lower I-ASR at these observed refusal rates (9.50% versus 13.67%). Increasing the benign fraction to 60% lowered both refusal rates (14.33% ORR and 21.00% I-ORR) and raised I-ASR to 17.02%.
Ablation and fixed-sampling results on DeepSeek-R1-Distill-Qwen-7B. ORR and I-ORR use XSTest. All rates use avg@4.
Thank you again for your constructive feedback, which helped us strengthen the empirical evaluation of DACPO. We would be happy to address any remaining questions. If the additional comparisons address your concerns, we would appreciate your reconsideration of the overall rating.