- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper argues that on-policy distillation (OPD) for safety mostly teaches the student to refuse, without teaching the policy distinction behind the refusal or how to recover from a harmful prefix. A diagnostic study supports this. Most early KL reduction sits in the first few tokens, boundary score lags behind safety error, teacher rationales are rarely covered by student rollouts, and recovery-state visits drop as refusal emerges. A 2×2 intervention then shows that policy rationales help boundary decisions and targeted starting states help recovery.
Based on this, DEEPEN combines four pieces:
- The student predicts structured policy labels (risk category, rule IDs, boundary attributes, response type) that condition each response segment.
- Training starts are sampled from a mixture of natural requests, boundary pairs, and harmful prefixes.
- Error-triggered, single-segment teacher handoffs are annealed to zero.
- Local distillation routes between RKL and top-k-union FKL by teacher–student overlap, with an FKL anchor to the initial student on benign states.
On four main models (ten configurations in total), DEEPEN improves BDA by 6.6 points and useful recovery by 18.1 points over an adapted PolicyAlign, with small capability drops.
Critical strengths
- The diagnosis is the most interesting part. Separating "what decision to make" from "whether the student ever visits the states where it matters" is a useful way to think about safety distillation, and the 2×2 rationale × state experiment backs it up.
- The control experiments are thorough. These include content controls (generic or shuffled rationales), predicted vs. oracle vs. wrong-field conditioning, matched-random starting states, extra natural rollouts, factorial designs for each pair of components, and a validation-ASR-matched comparison. It's clear the authors tried to rule out "it just refuses more."
- The evaluation is broad. It covers many safety, over-refusal, multilingual, and capability benchmarks across ten model configurations, and reports training cost and deployment latency.
Critical weaknesses
1. The paper is very hard to read. The main text is dense and hedged almost sentence by sentence. Sections 5.3–5.5 are essentially one sentence each, pointing to figures and appendices. The method has many interacting parts (label interface, three-way state mixture, replay pool, eligibility rubric, handoff schedule, overlap routing, tail aggregation, benign anchor), and key details live in a 40-page appendix. As a reviewer, I found it hard to tell what is essential and what the final recipe actually is.
2. Uncertainty and how some numbers were produced. All intervals are across request cohorts, not training seeds, so each configuration seems to come from a single training run. Some tables also look derived rather than measured:
- In Table 30, Qwen3-4B and Qwen3.5-4B have identical GPU-hours for every method, and so do Qwen3-8B and Llama 3.1 8B.
- Table 32 lists utilization factors, decode rates, and a lognormal length model, which suggests cost and possibly latency come from an accounting model.
- "Invalid recovery" is identical across all six methods for each model. Please clarify which numbers are measured and which are modeled.
3. It's unclear how much of the gain comes from extra data rather than the method. DEEPEN gets boundary pairs, harmful-prefix starts, two teacher annotations per checkpoint, and handoffs. The factorials show the prefix starts alone add ~18 points of recovery. It's not clear whether PA* and Relay* get the same boundary and prefix starting states. Without that, a large part of the improvement over PA* could be the data mixture. A natural baseline is PA* (or plain RKL) with the same critical-state mixture, plus a simple SFT baseline on boundary pairs and prefix-recovery data in the style of Qi et al. (2025). DEEPEN also uses ~1.5× the GPU-hours of PA*. In the compute-matched comparison (Table 11), it has the highest ASR of all methods (11.0 vs. 7.5–10.5).
4. The deployment interface and the trade-off in direct safety. At inference, the student emits hidden policy labels every B tokens with context rebuilds, adding 1.3–2.2× latency. This changes the model's interface relative to the baselines. Direct HarmBench ASR is slightly worse than PA* on three of four main models. The label channel is also a new attack surface. Boundary attributes like "authorization" and "intent" are inferred from the prompt, so an attacker who simply claims authorization could push the label toward "answer." This is not tested. Prefill-20 uses fixed prefixes, and there is no adaptive attack against the label predictor.
Minor: the teacher is the same checkpoint with a privileged policy document, so the method is closer to self-distillation and depends on a fixed rule ontology. Generalizing to policy changes or new rules isn't really tested. The theory in Appendices A and C mostly restates definitions, such as "more constraints shrink the hypothesis set," and adds little.
Questions for authors
- Are the main results from single training runs? Can you report variance across training seeds for at least the main comparison with PA*?
- Which cost and latency numbers (Tables 30–35) are measured, and which come from the accounting model in Table 32? Why is invalid recovery identical across methods?
- Do PA*, Relay*, RKL, and JSD receive the same boundary-pair and harmful-prefix starting states? If not, how does PA* with the same state mixture compare to DEEPEN?
- How robust is the label channel to prompts that falsely claim authorization or benign intent, or that otherwise target the predicted label?
- Can you give a minimal version of DEEPEN (e.g., labels plus state mixture, without handoff and routing) so readers can see what is essential?