acceptodds
Under review as a conference paper at ICLR 2027

Beyond Refusal: Policy-Grounded On-Policy Distillation for Safety Decisions and Recovery

Abstract

Safety distillation can teach a language model to refuse without teaching it the policy distinction behind the refusal—or how to recover once generation has entered a harmful trajectory. We study this gap in on-policy distillation (OPD) through representation readouts, trajectory diagnostics, and budget-matched interventions. Rule and boundary generalization lag behind refusal, while recovery states become less frequent in student rollouts as refusals emerge; explicit policy supervision improves boundary decisions, and targeted state coverage further improves recovery. Motivated by these findings, we introduce DEEPEN, a policy-grounded OPD framework that makes safety judgments explicit and restores access to decision-critical states. DEEPEN predicts structured safety labels to condition generation, trains on boundary pairs and harmful-prefix starts, and uses brief teacher handoffs triggered by policy errors to supply a corrective segment before returning control to the student. Overlap-aware token-level distillation reinforces local corrections, while handoffs anneal to zero for student-only deployment. Across four preselected representatives from distinct model families under matched response-token budgets, DEEPEN improves mean boundary score (BDA) by 6.6 percentage points and useful recovery by 18.1 points over an adapted PolicyAlign baseline; the largest task-wise capability decrease is 4.5 points relative to the initial student. These results support treating safety distillation as a joint problem of policy-decision learning and safety-critical state coverage, rather than refusal imitation alone.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety+OPD

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xpDBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that on-policy distillation (OPD) for safety mostly teaches the student to refuse, without teaching the policy distinction behind the refusal or how to recover from a harmful prefix. A diagnostic study supports this. Most early KL reduction sits in the first few tokens, boundary score lags behind safety error, teacher rationales are rarely covered by student rollouts, and recovery-state visits drop as refusal emerges. A 2×2 intervention then shows that policy rationales help boundary decisions and targeted starting states help recovery.

Based on this, DEEPEN combines four pieces:

  • The student predicts structured policy labels (risk category, rule IDs, boundary attributes, response type) that condition each response segment.
  • Training starts are sampled from a mixture of natural requests, boundary pairs, and harmful prefixes.
  • Error-triggered, single-segment teacher handoffs are annealed to zero.
  • Local distillation routes between RKL and top-k-union FKL by teacher–student overlap, with an FKL anchor to the initial student on benign states.

On four main models (ten configurations in total), DEEPEN improves BDA by 6.6 points and useful recovery by 18.1 points over an adapted PolicyAlign, with small capability drops.

Critical strengths
  • The diagnosis is the most interesting part. Separating "what decision to make" from "whether the student ever visits the states where it matters" is a useful way to think about safety distillation, and the 2×2 rationale × state experiment backs it up.
  • The control experiments are thorough. These include content controls (generic or shuffled rationales), predicted vs. oracle vs. wrong-field conditioning, matched-random starting states, extra natural rollouts, factorial designs for each pair of components, and a validation-ASR-matched comparison. It's clear the authors tried to rule out "it just refuses more."
  • The evaluation is broad. It covers many safety, over-refusal, multilingual, and capability benchmarks across ten model configurations, and reports training cost and deployment latency.
Critical weaknesses

1. The paper is very hard to read. The main text is dense and hedged almost sentence by sentence. Sections 5.3–5.5 are essentially one sentence each, pointing to figures and appendices. The method has many interacting parts (label interface, three-way state mixture, replay pool, eligibility rubric, handoff schedule, overlap routing, tail aggregation, benign anchor), and key details live in a 40-page appendix. As a reviewer, I found it hard to tell what is essential and what the final recipe actually is.

2. Uncertainty and how some numbers were produced. All intervals are across request cohorts, not training seeds, so each configuration seems to come from a single training run. Some tables also look derived rather than measured:

  • In Table 30, Qwen3-4B and Qwen3.5-4B have identical GPU-hours for every method, and so do Qwen3-8B and Llama 3.1 8B.
  • Table 32 lists utilization factors, decode rates, and a lognormal length model, which suggests cost and possibly latency come from an accounting model.
  • "Invalid recovery" is identical across all six methods for each model. Please clarify which numbers are measured and which are modeled.

3. It's unclear how much of the gain comes from extra data rather than the method. DEEPEN gets boundary pairs, harmful-prefix starts, two teacher annotations per checkpoint, and handoffs. The factorials show the prefix starts alone add ~18 points of recovery. It's not clear whether PA* and Relay* get the same boundary and prefix starting states. Without that, a large part of the improvement over PA* could be the data mixture. A natural baseline is PA* (or plain RKL) with the same critical-state mixture, plus a simple SFT baseline on boundary pairs and prefix-recovery data in the style of Qi et al. (2025). DEEPEN also uses ~1.5× the GPU-hours of PA*. In the compute-matched comparison (Table 11), it has the highest ASR of all methods (11.0 vs. 7.5–10.5).

4. The deployment interface and the trade-off in direct safety. At inference, the student emits hidden policy labels every B tokens with context rebuilds, adding 1.3–2.2× latency. This changes the model's interface relative to the baselines. Direct HarmBench ASR is slightly worse than PA* on three of four main models. The label channel is also a new attack surface. Boundary attributes like "authorization" and "intent" are inferred from the prompt, so an attacker who simply claims authorization could push the label toward "answer." This is not tested. Prefill-20 uses fixed prefixes, and there is no adaptive attack against the label predictor.

Minor: the teacher is the same checkpoint with a privileged policy document, so the method is closer to self-distillation and depends on a fixed rule ontology. Generalizing to policy changes or new rules isn't really tested. The theory in Appendices A and C mostly restates definitions, such as "more constraints shrink the hypothesis set," and adds little.

Questions for authors
  1. Are the main results from single training runs? Can you report variance across training seeds for at least the main comparison with PA*?
  2. Which cost and latency numbers (Tables 30–35) are measured, and which come from the accounting model in Table 32? Why is invalid recovery identical across methods?
  3. Do PA*, Relay*, RKL, and JSD receive the same boundary-pair and harmful-prefix starting states? If not, how does PA* with the same state mixture compare to DEEPEN?
  4. How robust is the label channel to prompts that falsely claim authorization or benign intent, or that otherwise target the predicted label?
  5. Can you give a minimal version of DEEPEN (e.g., labels plus state mixture, without handoff and routing) so readers can see what is essential?
Reviewer_dV79OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DEEPEN, a safety OPD framework combining structured policy supervision, critical-state sampling, teacher handoffs, and adaptive KL routing. It aims to improve the student's ability to distinguish harmful from benign requests and to recover from unsafe generation trajectories.

Critical strengths
  1. The paper studies an important problem of safety alignment beyond standard refusal behavior, with analysis of policy generalization and recovery-state coverage.
  2. The method achieves consistent improvements across model families, supported by extensive evaluations and compute-matched comparisons.
Critical weaknesses

W1. Limited new insight from the diagnostic analysis. Prior work on shallow safety alignment and harmful-prefix recovery has already highlighted related limitations. The analysis offers limited new insights beyond characterizing these known failure modes in the OPD setting.

W2. Limited methodological novelty and necessity. DEEPEN combines several largely established techniques without a distinct algorithmic contribution. The ablations show that some components already deliver substantial gains, while the value of combining all components remains insufficiently justified.

W3. Excessive complexity relative to the demonstrated benefits. The numerous interacting components introduce approximately 50% higher reported training costs and substantial inference latency. A clearer justification of this complexity–benefit trade-off is needed.

Questions for authors

Please address the concerns and clarification requests in W1–W3 under Critical Weaknesses.

Reviewer_WgiyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I find the introduction a bit hard to read.

  • "early refusal can bypass states that require recovery" What does "recovery" mean here?
  • "Refusal improves faster than rule and boundary generalization" what is "rule and boundary generalization"? What does "refusal improves fast" mean?
  • "while recovery-state visits decline as refusal emerges" What are "recovery-state visits"?
  • "explicit policy rationales improve boundary decisions and targeted state coverage further improves recovery." what are "explicit policy rationales"? what are "boundary decisions"? What is "targeted state coverage"?
  • "These findings motivate supervising both what safety judgment to make and how to act on it along a continuation." What is a "continuation"?

I was already lost after reading the first 3 paragraphs of the introduction and had to feed this paper to an AI. Here is my current understanding:

  • In Qi et al., it was highlighted that it is important for the models to learn how to refuse to generate further ("recovery") after outputting some harmful text. But in OPD, the student learns to refuse a harmful request immediately after seeing the request, which makes it hard to learn the recovery behavior.
  • To make the alignment deeper, the authors propose DEEPEN: it consists of (1) structured Policy Supervision; (2) critical-state sampling and teacher intervention; (3) teacher handoff; (4) distillation objectives and joint optimization.
Critical strengths

/

Critical weaknesses
  • The paper is so poorly written that only AI can understand it and explain it to me. In particular, the paper sometimes refers to one concept by more than 5 names. Without AI, I cannot understand this paper at all, even though I'm familiar with AI safety research. I'm 100% sure that reading this paper is a waste of reviewers' time. I don't believe that this writing quality is good enough for publication.
  • The main text of the paper does not contain sufficient information about the experimental setup. It is important to understand what each benchmark/dataset does and how each hyperparameter is picked, but I can hardly find any information about them.
Questions for authors

/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.