acceptodds
Under review as a conference paper at ICLR 2027

DACPO: Drift-Adaptive Constrained Policy Optimization for Cue-Free Safety Recovery in Large Reasoning Models

Abstract

Large reasoning models (LRMs) can reproduce safety corrections from supervised traces, yet this does not ensure that they will recognize and reverse an unfamiliar adversarial shift in their own reasoning. We call the discrepancy between cue-conditioned correction competence and cue-free self-correction the correction activation gap. We introduce Drift-Adaptive Constrained Policy Optimization (DACPO), a two-phase framework that combines supervised fine-tuning (SFT) and reinforcement learning (RL) for robust recovery from adversarial reasoning drift. The first phase uses SFT to establish a safety correction prior from adversarial correction trajectories. The second phase removes target corrections and uses RL to optimize cue-free recovery from policy-dependent worst-tail drift states. DACPO couples conditional value-at-risk state selection with trajectory-level recovery rewards and a persistent archive of newly discovered and historical failures, while an over-refusal constraint prevents safety gains from collapsing into indiscriminate refusal. Experiments across multiple LRMs and benchmarks show improved recovery in both directions while preserving direct safety and general reasoning utility. Together, these findings identify autonomous correction activation, rather than correction imitation alone, as a central requirement for robust reasoning alignment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Reasoning Models, Safety Alignment, Self-Correction, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Tet8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DACPO for recovering from perturbed reasoning states without an explicit correction cue. It combines supervised correction training with recovery-focused GRPO, using difficult-state selection, failure replay, and adaptive harmful/benign batch mixing. Experiments across four backbones show improvements in both direct and injected safety evaluations, with additional analyses of cue dependence, over-refusal, mathematical accuracy, and individual components.

Critical strengths
  • Clear problem formulation. The paper distinguishes initial safety from recovery after reasoning drift and evaluates cue dependence using matched perturbed states. This provides a useful perspective on safety beyond responses to intact prompts.

  • Consistent empirical improvements. DACPO achieves the lowest or tied-lowest direct and injected ASR across the evaluated backbone–benchmark combinations.

Critical weaknesses
  • The motivation for prioritizing recovery over prevention is insufficiently justified. While DACPO improves both direct and injected ASR, the paper does not establish why training models to recover from unsafe reasoning is preferable to strengthening their ability to avoid it. Injected evaluation forces models into perturbed states regardless of how effectively they prevent such states. A budget-matched comparison with training that aims to avoid unsafe reasoning in the first place would clarify the additional value of recovery-focused training.

  • The adaptive machinery is not sufficiently justified over simple recovery RL. The component ablations are useful, but do not establish an advantage over budget-matched recovery GRPO with fixed drift sampling and a tuned fixed harmful/benign mixture. This comparison would clarify whether the added complexity and probing cost are necessary.

Questions for authors

Q1. Under matched initialization, data, and training budgets, how does DACPO compare with safety training focused on preventing unsafe reasoning rather than recovering from it?

Q2. How does DACPO compare with simple recovery GRPO using fixed sampling and a tuned fixed harmful/benign mixture? What mixture replaces the controller in its ablation?

4 comments

Response to Reviewer

Thank you for asking whether recovery training adds value beyond prevention and how DACPO compares with fixed-sampling recovery GRPO.

Q1. Comparison with prevention-focused training

We compared DACPO with SAFEPATH and ThinkSafe adaptations on DeepSeek-R1-Distill-Qwen-7B. SafeChain, from the submitted Table 1, provides an additional baseline under the same evaluation protocol. DACPO had lower ASR and I-ASR than all three baselines on each benchmark. The unweighted mean ASR and I-ASR were 3.38% and 6.72% for DACPO, 30.30% and 36.29% for SAFEPATH, and 25.48% and 31.36% for ThinkSafe. Against ThinkSafe, the strongest baseline on both means, the gap was already 22.10 percentage points on intact prompts.

On harmful prompts, SAFEPATH supervises a short primer at the start of reasoning. ThinkSafe fine-tunes on full trajectories that pass a safety filter after refusal steering. DACPO probes the current policy to find difficult perturbed states and trains the model to continue safely from them. This difference in training exposure offers one explanation for the performance gap on shared perturbed prefixes. Direct ASR reflects both prevention and recovery during generation. The shared-prefix result supports training for recovery alongside prevention because it measures safety after all models have reached the same perturbed state. Direct safety and recovery from harmful drift on DeepSeek-R1-Distill-Qwen-7B. Both ASR and I-ASR use avg@4. The first three benchmarks evaluate harmful prompts. SafeUnlearning and WJ-AdvHarm evaluate jailbreak prompts.

MethodHarmBenchStrongREJECTWJ-VaniHarmSafeUnlearningWJ-AdvHarm
ASR ↓I-ASR ↓ASR ↓I-ASR ↓ASR ↓I-ASR ↓ASR ↓I-ASR ↓ASR ↓I-ASR ↓
DACPO2.004.502.885.112.673.674.0013.005.337.33
SAFEPATH adaptation36.6742.2038.0244.9526.3332.7227.8033.1022.6728.50
ThinkSafe adaptation30.3337.8025.6731.5025.5030.6721.8829.3024.0027.54
SafeChain (submitted Table 1)35.5044.0036.1038.3423.0029.3340.6748.0024.0029.00

Q2. Comparison with fixed-sampling recovery GRPO

Table 5 controller ablation. We replaced adaptive batch mixing with a fixed 8:2 ratio of harmful to benign state groups throughout Stage II and retained adaptive drift-state selection and failure replay.

Fixed-sampling recovery GRPO. We drew states from a frozen pool built from the initial drift archive and kept the correction-SFT checkpoint, source prompt pool, recovery rewards, and GRPO settings shared with DACPO. Of the fixed-sampling settings reported below, 6:4 was closest to DACPO on both XSTest refusal rates (16.10% ORR and 21.55% I-ORR versus 15.40% and 21.80%). DACPO achieved 4.17 percentage points lower I-ASR at these observed refusal rates (9.50% versus 13.67%). Increasing the benign fraction to 60% lowered both refusal rates (14.33% ORR and 21.00% I-ORR) and raised I-ASR to 17.02%.

Ablation and fixed-sampling results on DeepSeek-R1-Distill-Qwen-7B. ORR and I-ORR use XSTest. All rates use avg@4.

VariantHarmful:benign state-group ratioASR ↓I-ASR ↓ORR ↓I-ORR ↓
Full DACPOAdaptive4.509.5015.4021.80
DACPO without controller (Table 5)8:27.7510.6623.5029.43
Fixed-sampling Recovery GRPO6:49.2413.6716.1021.55
Fixed-sampling Recovery GRPO4:613.9917.0214.3321.00

Thank you again for your constructive feedback, which helped us strengthen the empirical evaluation of DACPO. We would be happy to address any remaining questions. If the additional comparisons address your concerns, we would appreciate your reconsideration of the overall rating.

Dear Reviewer Tet8,

Thank you again for your thoughtful review. As the discussion period is nearing its end, we wanted to follow up on our response. If you have a chance to take a look, we would appreciate hearing whether our clarifications address your concerns. We would be happy to answer any remaining questions. Thank you for your time and consideration.

We sincerely look forward to your feedback.

Thank you for your response. The additional experiments and clarification well addressed my concerns and questions. I have raised my score accordingly.

Thank you for taking the time to review our response and reconsider your score. We’re glad the additional experiments and clarifications addressed your concerns. Your feedback helped us improve the paper, and we appreciate your time throughout the review process.

Reviewer_9HruOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2(−1)
Read review
Paper summary

This paper proposes DACPO, a two-stage framework for cue-free recovery from harmful or over-cautious reasoning drift. It first uses supervised fine-tuning to learn correction patterns, then applies reinforcement learning with hard-state selection, trajectory-level rewards, failure replay, and over-refusal control. Experiments across four language-model backbones and multiple safety benchmarks show improved direct and injected safety, while maintaining relatively stable benign refusal and mathematical reasoning performance.

Critical strengths

1.The problem is clearly defined and meaningful.

2.The paper is well structured and logically presented.

3.The empirical evaluation is fairly broad, and the results are strong.

Critical weaknesses

1. Limited novelty. DACPO combines supervised correction initialization, RL-based self-correction, hard-state selection, trajectory-level rewards, failure replay, but most of these ingredients follow established paradigms. As a result, the method appears to contribute mainly by combining these ingredients for safety drift recovery, rather than by introducing fundamentally new mechanisms.

2. Limited generalization evidence. The model may exploit lexical or template-specific cues in the injected drifts, leading to overestimated benchmark performance and weak transfer to differently phrased or naturally occurring drifts.

Questions for authors
  1. What is DACPO’s unique contribution beyond combining existing paradigms?

  2. How do the authors rule out reliance on lexical or template-specific drift cues, and how does DACPO perform on paraphrased or naturally occurring drifts?

  3. Could DACPO’s smaller activation gap reflect a floor effect from its lower overall ASR rather than reduced cue dependence? Can the authors disentangle these effects?

I would consider raising my score if the authors’ responses to these questions address my concerns.

5 comments

Response to Reviewer

Thank you for recognizing the importance of cue-free recovery and for asking about the method, generalization, and interpretation of the activation gap.

1. DACPO's Contribution

DACPO's contribution is a way to expand where a model can apply a correction learned by imitation, using its own cue-free failures to select what it practices next. We curate 1,000 trajectories that show the transition from a reasoning error to its repair in both directions, from harmful compliance toward a safe answer and from premature refusal toward a helpful answer. SFT gives the model these two corrective behaviors by imitation.

The SFT checkpoint both initializes the RL policy and supplies the reasoning chains used to construct its training states. This gives RL a model that has learned how to correct itself and a way to test where it can already do so without a cue. DACPO’s distinctive step is to use the current policy’s own recovery outcomes to decide what it should practice next. RL prioritizes states where the model still answers unsafely or refuses a benign request, and newly observed failures inform subsequent rounds. In this way, SFT teaches the corrective behavior, and RL expands the range of states in which the model can apply it autonomously.

2. Generalization beyond drift templates and to model-generated failures

Thank you for your question about generalization. We address reliance on template or lexical cues and naturally occurring reasoning drift separately.

(1) Reliance on template or lexical cues. We use disjoint prompts and drift templates for training and evaluation, and inject drift at three different positions along the reasoning chain. Evaluation therefore requires cue-free recovery on unseen prompts and drift expressions across different reasoning contexts. Table 1 shows that DACPO achieves lower I-ASR than Self-ReSET in all 20 comparisons spanning four backbones and five benchmarks, demonstrating that its recovery advantage transfers to held-out drift templates. Together, these results show that DACPO's safety gains extend across both evaluation with unseen injected templates and ordinary generation without injection, beyond recognizing drift expressions encountered during training.

(2) Naturally occurring reasoning drift. DACPO directly uses failures arising during the model's own generation as training material. The initial drift archive is constructed from drift segments preceding the correction in Stage I correction trajectories. During Stage II, we further extract and filter new drift segments from the current policy's failed continuations and add them to the archive for subsequent training. As the policy is updated, the failure states covered by training are continuously updated as well, allowing the model to practice recovery from harmful reasoning or unwarranted refusal states that arise in its own continuations. This is a key mechanism through which DACPO extends its correction capability to the model's own failure modes. Empirically, the direct evaluation in Table 1 applies no drift injection and allows the model to generate autonomously from the original prompts; DACPO also achieves lower ASR under these conditions. The use of the model's own failures during training and the observed safety gains without injection provide complementary support for DACPO's effectiveness in autonomous generation. ift.

3. Could the smaller activation gap be a floor effect?

Yes. Thank you for raising the possibility of a floor effect. In Table 2, DACPO's cue-free ASR is 4.50%, which bounds the largest possible absolute reduction from a cue.

We also evaluated both methods on 100 shared perturbed prefixes selected for high cue-free failure rates under DACPO in a separate screening run. We estimated ASR with and without the same correction cue using fresh continuations. Adding the cue reduced ASR from 23.25% to 7.50% for AdvChain and from 14.00% to 5.25% for DACPO. The activation gaps were 15.75 and 8.75 percentage points, respectively.

MethodWith cue ASR ↓Without cue ASR ↓Activation gap ↓
AdvChain7.5023.2515.75
DACPO5.2514.008.75

The smaller activation gap persists on the 100 cases selected to be difficult for DACPO, even though its cue-conditioned ASR remains at 5.25%. DACPO's gap is 7.00 percentage points smaller than AdvChain's on these shared cases. The cue also reduces DACPO's ASR by a smaller proportion of its cue-free rate (62.5% versus 67.7% for AdvChain). Together, these results show that the smaller gap persists above the zero-ASR floor and that DACPO retains a 9.25-point advantage in cue-free ASR on the hard subset.

Thank you again for your constructive feedback. We would be happy to address any remaining questions. If these clarifications address your concerns, we would appreciate your reconsideration of the overall rating.

Dear Reviewer 9Hru,

Thank you again for your thoughtful review. As the discussion period is nearing its end, we wanted to follow up on our response. If you have a chance to take a look, we would appreciate hearing whether our clarifications address your concerns. We would be happy to answer any remaining questions. Thank you for your time and consideration.

We sincerely look forward to your feedback.

Thank you for your response. The additional experiments and clarification well addressed my concerns and questions.

Reviewer_RgQNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies recovery after a large reasoning model's intermediate reasoning prefix is perturbed, distinguishing cue-conditioned correction from cue-free continuation. DACPO first learns a bidirectional correction prior from 1,000 teacher-generated trajectories. It then uses GRPO to train recovery from difficult injected states, combining CVaR-inspired tail selection, sentence-level safety rewards, a persistent failure archive, and an adaptive harmful/benign batch mixture guided by an over-refusal target. Evaluation covers four backbones and five harmful or jailbreak benchmarks, with additional benign-refusal and mathematical-reasoning evaluations on DeepSeek backbones. The paper reports lower direct and injected attack success rates, a smaller correction activation gap on a 100-prompt harmful subset, and improved recovery from over-cautious drift on the evaluated 7B model. The conclusions are explicitly scoped to controlled prefix injection rather than naturally occurring reasoning failures.

Critical strengths
  1. Substantial and consistent reported recovery improvements. Table 1 evaluates four backbones across five harmful/jailbreak benchmarks. DACPO attains the lowest or tied-lowest reported I-ASR throughout, reducing the unweighted mean I-ASR from 14.17% for AdvChain to 6.75%. The improvement is not restricted to one backbone or benchmark, which makes the empirical recovery result consequential. Direct ASR also generally improves, providing evidence that the recovery training does not merely optimize the injected setting at the expense of ordinary prompt safety.

  2. A useful recovery formulation with checks against indiscriminate refusal. The matched cue/no-cue experiment reports both absolute failure rates and their difference, avoiding the misleading interpretation that any small activation gap implies successful recovery (Section 4.3, Table 2). On DeepSeek-R1-Distill-Qwen-7B, Table 3 reports near-base direct refusal rates alongside lower injected refusal rates. This is an important complement to harmful-request safety evaluation and supports the usefulness of the bidirectional training objective within the tested setting.

Critical weaknesses
  1. W1. The central adaptive-training comparison is insufficiently specified to audit or reproduce. Sections 3.3–3.5, 4.1, and Appendix A describe the framework but do not specify the seed drift archive and held-out drift construction, concrete archive admission/filtering rules, or several operative settings such as the archive size, mining threshold, and controller initialization/bounds. The exact evaluation subsets and aggregation over injection depths are also unclear. Baselines are said to use matched prompts, decoding, and judges, but their training data, optimization budgets, and tuning procedures are not reported. Similarly, Table 5 does not explain what replaces each removed component, including whether disabling the controller keeps a matched fixed benign fraction. Consequently, the reported method-level gains cannot yet be cleanly separated from differences in data exposure, search effort, or batch composition. This is particularly consequential because adaptive state selection and replay are central contributions.

  2. W2. The principal safety evaluation is not sufficiently independent of the training signal. Llama-Guard-3-8B supplies training safety scores and the final-answer safety metric (Section 4.1, lines 318–323). The reported manual check comprises only 50 harmful and 50 benign responses, with 98/100 pooled agreement and no breakdown by method, injected/direct condition, or failure type (lines 360–366). This is useful preliminary validation but does not establish that DACPO's advantage persists under an independent assessment of harmful compliance, particularly on outputs selected or optimized using the same judge. The benign evidence also counts only full refusals, although partial refusals are classified, so near-base ORR alone does not establish preserved helpfulness. An independent, stratified check of the key comparisons, including partial-refusal outcomes, would materially strengthen the central safety/recovery conclusion.

Questions for authors
  1. Protocol clarification, W1. What are the sources and construction procedures for the seed archive and held-out harmful/benign drift templates, how many are used, and what checks enforce their separation? Are the exact same frozen base prefixes and injected states used for every compared method? Please specify the evaluation sample counts and aggregation over injection depths.
  2. Comparison clarification, W1. How were AdvChain and Self-ReSET instantiated and tuned, and how do their training examples, generated tokens, and optimization budgets compare with DACPO? For the selection, archive, and controller ablations, what sampling/replay rules replace the removed component, and is benign exposure matched when the controller is disabled?
  3. Validation request, W2. How were the 100 manually checked responses sampled across methods and conditions? Can the key DACPO-versus-baseline comparison be assessed by an independent judge or a blinded, stratified human audit, with harmful-compliance errors reported separately from benign-refusal errors? For Table 3, does the conclusion remain when partial refusals or answer usefulness are included?
4 comments

Response to Reviewer

Thank you for your positive assessment of our work. We agree that the original submission did not describe the experimental procedures in enough detail for reproduction. We will provide a fuller account of the training and evaluation protocols in the revised manuscript.

Q1. Drift construction

Seed archive. We initialize the Stage II archive with filtered drift segments from the Stage I correction trajectories. We extract the span where the reasoning begins to drift toward harmful compliance or unwarranted refusal, ending the span before the explicit correction. During Stage II, we add filtered drift segments mined from failed continuations. The archive uses a 50% elite quota and a UCB coefficient of 1.4. The controller uses an over-refusal target of 0.20 and an update step size of 0.05.

Held-out evaluation. Training and evaluation use disjoint prompts and drift templates. For harmful injection, we insert a held-out drift at 25%, 50%, or 75% of a frozen correction-policy reasoning chain and generate four independent cue-free continuations. Table 1 matches prompts, decoding settings, and judges across methods. The 100 harmful prompts in Table 2 refer specifically to the matched cue/no-cue comparison.

Q2. Baseline reproduction and ablation settings

AdvChain and Self-ReSET. We reproduced both baselines on the corresponding Table 1 backbones following their respective papers. AdvChain uses supervised fine-tuning on correction demonstrations. Self-ReSET uses recovery RL from unsafe prefixes generated by the policy. The published configurations use 1,000 demonstrations and five SFT epochs for AdvChain, and 3,000 prompts, three epochs, and 16 rollouts per sampled prompt for Self-ReSET. Table 1 evaluates the resulting methods under matched evaluation conditions. Mean I-ASR across the 20 settings is 6.75% for DACPO and 14.17% for AdvChain. The comparison is at the method level, with each baseline following its own training recipe.

Table 5. Removing the controller fixes the Stage II harmful-to-benign state-group ratio at 8:2, while retaining adaptive state selection and failure replay. The fixed setting allocates 20% of groups to benign states. The full method adjusts this fraction during training, so the two runs' cumulative benign exposure depends on the realized group counts. With the controller removed, I-ASR rises from 9.50% to 10.66% and XSTest I-ORR rises from 21.80% to 29.43%.

Q3. Manual audit, independent judging, and partial refusals

We randomly sampled 50 responses to harmful prompts and 50 responses to benign prompts for manual annotation. Human and automated labels agreed on 98 of the 100 sampled responses.When we count both partial and full refusals, DACPO achieves similar reductions in refusal rates relative to the base model on XSTest and WJ-AdvBenign after drift injection.

Thank you for the detailed feedback. Your questions helped us clarify the evaluation and the scope of the results. We would be happy to discuss any remaining questions.

The authors’ response has largely addressed my concerns, and I have no further questions. I will maintain my current score. While I do not believe the paper warrants the highest rating, I remain inclined to recommend acceptance.

Additionally, Figures 1 and 2 appear somewhat blurry and errors (Qwen logo at the bottom Fig2). The authors should manually redraw them at a higher resolution rather than rely on AI-generated images, ensuring that all text and visual elements are clear and legible.

Thank you for pointing out the legibility issues and the extra Qwen logo at the bottom of Figure 2. We will remove it, improve the clarity of the text and visual elements in both figures, and upload the revised manuscript shortly.

Comments on the paper

1 comment

Dear Area Chair and Reviewers, thank you for your feedback. We have uploaded a revised PDF with two changes. In Figure 2 (page 4), we removed the duplicate Qwen logo from the legend. We also added Appendix C (pages 14–16), which brings together the additional baseline comparisons (C.1), the results on held-out drift templates and the analysis of the activation gap (C.2), and the reproducibility details and manual evaluation (C.3). Thank you for taking the time to review the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.