acceptodds
Under review as a conference paper at ICLR 2027

DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision

Abstract

On-policy distillation (OPD) provides dense token-level supervision on student rollouts, but this supervision can become unreliable when the student and teacher diverge, either because of a large initial policy gap or compounding errors in long-horizon reasoning and tool use. We introduce Divergence-Triggered Dynamic Distillation (DTDD), an adaptive OPD framework that routes supervision between standard OPD and teacher-generated recovery using segment-level student–teacher divergence. When divergence triggers an intervention during reasoning, the student prefix continues to receive standard OPD supervision. The teacher then generates a recovery suffix, which the student learns through behavior cloning, an online SFT-style loss. We evaluate DTDD in three regimes: long-horizon reasoning, distillation from a base initialization without offline SFT, and multi-turn agents. We also compare alternative trigger designs and teacher-intervention rates. DTDD reduces the post-warmup peak gradient norm by 8.6× and, from a base initialization, reaches 21.3% held-out math accuracy in 2.6× fewer optimizer updates than OPD; both methods finish near 22%. It improves best-observed performance by 4.8 and 4.1 percentage points on ALFWorld and agentic code, respectively. Ablations show that segment-level triggers outperform token-level triggers. Behavior cloning is also more effective than the standard OPD objective for training teacher-generated recoveries. Across 23 experimental arms, takeover tends to help when the teacher substantially outperforms the OPD-trained student, provided that the teacher can recover from student-visited states, the student can learn those recoveries, and teacher intervention is neither excessive nor insufficient.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, large language models, reasoning, multi-turn agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5QsZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

the paper tackles a real pain point in on-policy distillation: the per-token teacher signal on student rollouts stops being useful once the student drifts away from the teacher .

DTDD adds a takeover rule on top of plain OPD: chop the rollout into fixed-length segments, compute an averaged nonnegative student/teacher discrepancy per segment, and the first segment crossing a threshold triggers a rewind — keep the student prefix, let the frozen teacher regenerate the suffix, then train the prefix with OPD and the teacher-written suffix with behavior cloning. no trigger means you get exactly OPD back.

Critical strengths

the mechanism is genuinely simple and it is very effective

Critical weaknesses

Table 3's headline is best-observed checkpoint picked every 20 updates, which is exactly what inflates a +4.8 claim.

the long-horizon contribution is also partly undercut by their own clipping control. and scope is narrow: students 0.5B-3B, teachers to 7B, budgets to 18K, only math plus two agent suites;

Questions for authors

see the weakness above.

1 comment

We thank the reviewer for the careful review. The ALFWorld LoRA gain persists at the final shared checkpoint, and the clipping comparison on DAPO shows where DTDD helps early in training.

W1. Checkpoint selection. In the original ALFWorld LoRA run, DTDD’s advantage was larger at the shared final checkpoint (+7.53 percentage points) than at the methods’ respective peaks (+4.83 points). Table 3 compares DTDD at update 140 (81.32%) with OPD at update 280 (76.49%). At update 400, DTDD reached 77.59% and OPD reached 70.06% (Table 6 in the submission, Table 18 in the revised manuscript).


W2. If clipping already stabilizes OPD, what does DTDD add?

Clipping stabilizes OPD, and DTDD also accelerates early learning. On the same DAPO seed, DTDD first exceeded 20% accuracy at update 60, compared with update 200 for both vanilla and clipped OPD (Table 17 in the revision). Vanilla and clipped OPD give teacher feedback along the path the student takes. When the student’s reasoning goes off track, DTDD lets the teacher continue from the student’s prefix and show it how to proceed. This matters early in training, when a base student may rarely generate a useful reasoning path on its own.

We tested clipping in the long-horizon stability experiments (Table 10 in the submission, §4.2 and Table 8 in the revision) and added a clipped OPD run to the DAPO comparison using the same base student, teacher, and rollout seed. At update 80, DTDD reached 21.34% accuracy, compared with 13.40% for clipped OPD and 13.99% for vanilla OPD. At update 120, DTDD remained ahead of both (§4.3 in the revision). At update 200, vanilla and clipped OPD scored 21.25% and 22.12% on seed 1, and DTDD scored 21.96% in the longer seed-0 run (Tables 16 and 17 in the revision).

Scope. Our experiments cover math reasoning, ALFWorld, and agentic coding, with students from 0.5B to 3B parameters, teachers up to 7B, and response budgets up to 18K tokens. We agree that testing whether the gains extend to larger students and teachers and to longer response budgets is an important next step. Given our compute budget, we leave that evaluation to future work.

Reviewer_HRTSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper introduces DTDD to improve distillation when the student and teacher diverge. It measures disagreement over student-generated segments, rewinds to the segment boundary when a threshold is crossed, and asks the teacher to continue. For reasoning, the student prefix receives OPD supervision and the teacher suffix receives behavior cloning (BC). Interactive tasks use action replacement or teacher takeover of later turns. Experiments cover long reasoning, distillation from a base model, and multi-turn agents. The main findings are more stable training, faster early learning, and higher scores in some configurations.

Critical strengths
  1. The problem is concrete and the method has practical value. Checking segments to decide when teacher continuation is needed addresses drift in student generation and slow early learning from a base model. Evaluation uses only the student, so the reported scores do not depend on teacher assistance at test time.
  2. The experiments cover several settings and include useful controls. Paired runs match initialization, teacher, prompts, generation budget, and optimizer. The paper also studies segment size, intervention frequency, and suffix objectives. The main base-model OPD/DTDD comparison includes two training seeds, which is more convincing than a single run (Tables 1 and 12).
  3. The appendices clearly report the limits of the results, including fixed-step comparisons, full curves, failures with excessive intervention or an unsuitable teacher, and a simple clipping control that also stabilizes training. In the 15K reasoning setting, the peak gradient falls from 12.02 to 1.40 while accuracy changes from 54.1% to 54.0%. This supports a stability benefit, rather than higher final accuracy, in that setting (Tables 9 and 10).
Critical weaknesses
  1. The distinction from close prior work is not clear enough. AdaSwitch already triggers teacher continuation using divergence [R1]. Section 4.5 of the July 2026 first version of DASH-OPD combines state-dependent switching with losses that depend on who generated the tokens [R2]. Guided-OPD also uses different objectives for student and teacher turns [R3]. These papers are cited; the concern is that the introduction describes them too broadly. DTDD may add segment averaging, rewinding, and different takeover scopes. At update 120, its margin over AdaSwitch is only 0.88 percentage points, with one AdaSwitch run (Table 1). A matched DASH-OPD or Relay-OPD comparison [R4] is missing.
  2. It is still unclear which design choice produces the gains. In Tables 12 and 13, the methods differ in when the teacher takes over, how often this happens, and how much text the teacher generates. Better DTDD results therefore do not by themselves show that choosing takeover times based on divergence is responsible. The BC versus ordinary distillation (KD) comparison has a similar issue: the KD candidates may omit the token actually generated by the teacher (Appendix D.3), so the two objectives do not use exactly the same information. More closely matched controls are needed to separate the effects of takeover timing and the training objective.
  3. The evidence is not yet sufficient to show that the gains are stable. Table 3 reports each method at its best training checkpoint, while the gap can shrink at a fixed training step. For example, the average code-task gain falls from 4.10 percentage points at the best checkpoints to 1.03 points at the training endpoint (Tables 3 and 18). Apart from the main base-model comparison with two training seeds, most settings have only one training run. Sampling multiple answers from the same model, as in average@16/8, does not show that training again would give a similar gain. The results support benefits in some settings, but do not yet establish their stability.
  4. Some training settings need clearer descriptions; this is a smaller concern. The main text describes BC for the teacher-generated continuation, but Equation (9) in Appendix B.2 shows that ALFWorld and WebShop use a different objective. This difference should be stated where those experiments are presented. The teacher model name should also be consistent: Tables 3/4 say Qwen3-4B, while Table 15 says Qwen3-4B-Base. Clarifying these points would make the experiments easier to understand.
Questions for authors
  1. Which design choices does DTDD add beyond DASH-OPD v1 Section 4.5, Guided-OPD, and AdaSwitch? Please make the differences explicit and prioritize one close baseline using the same models, environment, and teacher budget.
  2. Can divergence triggering be compared with random takeover while keeping the amount of teacher-generated text and the number of takeovers similar? For BC versus KD, can both use the same generated sequences, with the teacher-generated token included among the KD candidates? These controls would help establish whether takeover timing and the training objective each contribute to the gains.
  3. Can the main methods be compared at the same training step, with uncertainty in their score differences estimated from existing per-problem outputs? If results from independent training runs are already available, please report them as well to help assess whether the gains are stable.
3 comments

We thank the reviewer for highlighting the need for closer controls.

W1/Q1

1. What does DTDD add beyond AdaSwitch, Guided-OPD, and DASH-OPD?

DTDD triggers on segment-average divergence and rewinds to the segment’s start, reducing spike sensitivity versus AdaSwitch while enabling teacher continuation from the student’s prefix before drift. With identical behavior-cloning loss, accuracy at update 120 was 20.83% versus 16.79% for token-spike triggering (Table 1).

Unlike Guided-OPD’s scheduled turns or DASH-OPD’s switching with accumulated drift/recovery evidence, DTDD replaces only high-divergence actions on ALFWorld/WebShop; the student immediately resumes from the new observation (§2).

2. Is the 0.88-point gain over AdaSwitch consistent across checkpoints?

Both DTDD seeds outperformed the AdaSwitch run at all four matched checkpoints (updates 60–120), with a mean margin of 0.88 percentage points at update 120 (Tables 1 and 13). Paired 95% CIs assess sensitivity to evaluation problem selection (§4.1; Appendix D.2, Table 15).

UpdateDTDD (2-seed mean)AdaSwitch (1 seed)Difference (pp)Paired 95% CI (pp)
6020.32%18.16%+2.16[+0.45, +4.05]
8021.47%19.57%+1.90[+0.42, +3.47]
10021.08%18.68%+2.40[+0.97, +3.89]
12020.83%19.95%+0.88[−0.02, +1.86]

The paired 95% confidence intervals were entirely above zero at three of the four checkpoints.

3. How does DTDD compare with DASH-OPD on the same ALFWorld setup?

With identical student/teacher, LoRA, and training seed, DTDD surpassed DASH-OPD at update 140 on Table 3’s ALFWorld sample: 81.32% vs. 79.46% macro success, with gains from unseen games (78.36% vs. 74.63%).

DTDD also used fewer teacher-generated tokens in the runs we measured. Over 140 training updates, DASH generated 479k teacher tokens (seed 42), compared with 168k and 156k for DTDD (seeds 43 and 44). The accuracy comparison above used seed 42 for both methods, with no fixed teacher-token budget (§4.4 and Appendix E.1).


W2/Q2

1. Does divergence-triggered takeover help the student learn faster than random takeover?

With similar teacher-token use (22M vs. 21M), DTDD reached roughly 20% accuracy in 40 updates versus 60 for random takeover, using fewer takeovers (7,669 vs. 9,376). From update 60 onward, both stayed around 20–22%. See §4.3 and Appendix Table 13 and its accompanying text for the setup and full-run costs.

2. Could missing teacher tokens explain the gap between BC and KD?

Missing teacher tokens are unlikely to explain the BC–KD gap. With the same segment trigger, BC reached 20.76% accuracy at update 120 versus 10.23% for KD (Appendix Table 13). To test the reviewer’s explanation, we evaluated both KD checkpoints on the same 128 saved teacher demonstrations (407,071 tokens). The student’s top 16 omitted the teacher’s token only 0.083% of the time at update 20 and 0.082% at update 80. This checks token coverage on shared demonstrations. We did not retrain BC and KD on identical sequences with the teacher token explicitly included (§4.1 and Appendix D.2).


W3/Q3. Do the gains persist at the same training step and across independent training runs?

To test whether the gains persist across training seeds, we added independent runs, bringing the totals to two seeds each for coding and DAPO math and three for ALFWorld LoRA. Time and compute limited replication to these three settings. The new table in Appendix B.4.1 reports per-task means and sample SDs at matched updates and at each method’s best checkpoint within the same budget.

At matched updates, DTDD had higher mean macro scores and lower across-seed SDs than OPD in all three settings. DTDD led in both DAPO seeds (mean gain 6.29 pp), in two of three ALFWorld LoRA seeds (1.45 pp), and in both coding seeds (3.86 pp).


W4. Teacher name and training objective.

We now consistently refer to the OpenThoughts3 teacher as Qwen3-4B. We also clarified in §4.4 that ALFWorld and WebShop apply the same clipped OPD objective to both student- and teacher-written actions. Each executed token is weighted by its log-probability under the teacher minus its log-probability under the rollout student (Eq. 9 in the submission, Eq. 8 in the revision). Math and agentic code use behavior cloning on teacher-written tokens.

Thank you for the detailed response and additional experiments. The matched-step and multi-seed results largely address my concerns about the reliability of the gains. The token-coverage analysis and clarification of the training objectives also resolve the corresponding concerns. However, I still consider the novelty limited: segment-level triggering, rewinding, and action-level replacement are useful refinements of existing teacher-takeover methods. I therefore raise my Soundness score from 1 to 2, while keeping my overall rating at 2 (weak rejection).

Thank you for revisiting our results and for raising the Soundness score. Regarding your remaining concern on novelty, we respectfully clarify why our contribution goes beyond incremental refinements of existing teacher-takeover rules:

1. A principled answer to when and why teacher takeover helps (rather than another heuristic).
Our central contribution is establishing when ordinary OPD already provides sufficient supervision and when teacher-generated continuations genuinely improve learning efficiency. Across 23 experimental arms in 13 settings (§4.5, Appendix F), our headroom analysis links takeover gains directly to the performance gap remaining between the OPD student and its teacher, and our boundary controls isolate three governing failure modes: teacher validity (WebShop), student learnability (0.5B ALFWorld), and excessive takeover dose (agentic code). Importantly, this same headroom framework explains the scale-dependent gains and regressions in published TCOD results (Appendix F.3), offering a general diagnostic for the broader teacher-takeover paradigm rather than a paper-specific observation.

2. Non-trivial synergy and practical impact in DTDD’s design.
Rather than combining interchangeable tweaks, DTDD’s three ingredients—segment-level divergence triggering, boundary rewinding, and role-split supervision (OPD on student prefixes, BC on teacher recoveries)—are tightly coupled. Replacing either the segment trigger (20.83% → 16.79%) or role-split BC (20.83% → 10.23%) sharply degrades base-model learning (Table 1). This unified mechanism also delivers substantial practical gains across regimes: it eliminates the two-stage offline SFT + OPD pipeline while matching accuracy at 1.9× less serial time and 55% fewer teacher tokens (Table 17), reduces long-horizon peak gradient norms by 8.6×/7.0×, and uses ~3× fewer teacher tokens than DASH-OPD on ALFWorld and ~40% fewer teacher tokens per step than AdaSwitch (Table 14).

Together, the mechanism and the 23-arm diagnostic provide both a concrete recipe and generalizable principles for supplementing OPD with teacher supervision. If these clarifications address your remaining concern on novelty, we would be grateful if you could reconsider your overall rating.

Reviewer_X51DOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper raised a method DTDD routes OPD using segment-level student–teacher divergence: when no trigger fires, the student rollout is retained; when a trigger fires, the method rolls back to the segment boundary and has the teacher regenerate the suffix. The student prefix uses OPD, while the teacher suffix uses behavior cloning. The paper covers long-sequence mathematics, online distillation from base initialization, ALFWorld, and agentic code, comparing segment/token triggers, AdaSwitch, rewind/restart, takeover dose, clipped OPD, and offline SFT→OPD.

Critical strengths
  • The method uses divergence to locate unreliable regions and provides coherent teacher recovery while retaining the on-policy prefix, making the design easier to interpret than uniform KD.
  • The high-dose, over-replacement, teacher-invalid, student-unlearnable, and loss-clamp controls avoid presenting the method as universally effective takeover.
Critical weaknesses
  • The paper explicitly states that several curves have only one training run, while average@k mainly reflects sampling variance; best-observed gains of 4.8/4.1 points cannot substitute for multi-seed results at fixed checkpoints, and the matched-update gains for full-model ALFWorld and code are limited.
  • In the 15K/18K stability experiments, a simple [-1,1] OPD clamp already removes the gradient storm; DTDD still requires additional teacher continuation, threshold tuning, and a random floor. Hand-set dose and heterogeneous hardware timing also hinder strict efficiency comparisons.
Questions for authors
  1. Can all headline results be reported again with at least three training seeds, preregistered checkpoint selection, and per-task paired intervals, while clearly separating best-observed from matched-update results?
  2. Under matched stability, teacher tokens, GPU-hours, and wall-clock time, do DTDD, [-1,1] clamp, ordinary OPD, and offline SFT→OPD still show an independent accuracy/cost advantage?
  3. Were the threshold and target takeover settings tuned using evaluation results? Can recovery quality and trigger precision/recall predict when takeover is effective, rather than relying only on divergence?
2 comments

We thank the reviewer for highlighting the value of retaining the student’s prefix. The added comparisons show that DTDD’s clearest advantage over clipped OPD is faster early learning on DAPO, and its gains over ordinary OPD also appear across training seeds.

W1/Q1. Do the gains hold across training seeds and checkpoint choices?

We added independent training runs for DTDD and OPD, bringing the total to two seeds each for DAPO math and coding and three for ALFWorld LoRA. The new table in Appendix B.4.1 (“Results across Training Seeds,” p. 19) reports per-task means and sample SDs at matched updates and at each method’s best checkpoint within the same training budget. We chose the budget windows retrospectively. Because time and compute limited replication to a subset of the main experiments, we report the detailed results in the appendix.

DTDD had higher mean macro scores than OPD in all three replicated settings under both checkpoint comparisons. At matched updates, DTDD led in both DAPO seeds (mean gain 6.29 pp), both coding seeds (3.86 pp), and two of three ALFWorld LoRA seeds (1.45 pp). Its macro scores also had lower sample SDs across seeds in all three settings at matched updates.


W2/Q2

1. If clipping already stabilizes OPD, what does DTDD add?

DTDD’s main advantage is faster early learning. When the student’s reasoning goes off track, the teacher continues from the student’s prefix and shows it how to proceed. This is especially useful early in training, when a base student rarely generates a useful reasoning path on its own. DTDD also suppresses late gradient spikes on long rollouts, a benefit that clipping shares (§4.2, Appendix Table 8).

Clipping alone cannot reproduce the early learning gain, we added a clipped OPD run using the same base student, teacher, and DAPO rollout seed. DTDD’s first evaluated checkpoint above 20% accuracy was update 60, compared with update 200 for both vanilla and clipped OPD. At the shared update-80 checkpoint, DTDD reached 21.34%, versus 13.99% for vanilla OPD and 13.40% for clipped OPD (§4.3, Appendix Table 17).

2. How much tuning do the takeover threshold and the random floor require?

Takeover threshold. We set the threshold once from initial training rollouts to target the desired takeover rate (App. B.1.1), holding it at 0.7 across all three response budgets in the stability experiments. At 18K, without retuning, teacher takeover fell from ~91% of rollouts initially (App. C.3) to ~10% by update 150, briefly rose around update 210 as divergence increased, and ended below 5% (submission Fig. 10f).

Random takeover floor. The same 5% probability across all three budgets lets the student occasionally learn from teacher continuations even when divergence stays below the threshold (App. B.1.1, C.3). To test whether late training stability depended on it, we ran an additional ablation continuing from the 18K run’s update-170 checkpoint with the floor disabled and divergence triggering active. Over updates 171–279, peak gradient norm was 0.51 with the floor off vs. 0.46 in the original run (floor on); mean held-out accuracy across the six subsequent checkpoints was only 0.05 percentage points lower with the floor off (App. C.3).

3. DTDD’s cost vs. offline SFT→OPD and ordinary OPD

DTDD and SFT→OPD learn from teacher text, generated online upon divergence and offline for SFT before OPD, respectively, so teacher-written tokens are comparable: 43M vs. 95M (55% fewer) at 21.34% vs. 21.55% DAPO accuracy. Measured serial times: 22.0 vs. ≈42.8 h on different hardware (submission Table 16).

Ordinary/clipped OPD use teacher scores on student reasoning, not teacher continuations, so this token metric is not comparable. Since update costs differ, revised Table 17 compares costs to the first evaluated checkpoint >20% accuracy. Ordinary OPD’s preceding checkpoint scored 18.84% at update 160.

MethodUpdateTeacher-written tokensSerial hoursGPUs
DTDD6032.8M15.714×L40S
SFT→OPD20 after SFT95M≈42.8A40/A100
Ordinary OPD200016.142×A100
Clipped OPD200020.214×L40S

SFT→OPD time includes generation, SFT, checkpoint merging, and OPD. Clipped OPD evaluations: updates 80/120/200.


Q3. Threshold selection and trigger utility

No. We set thresholds and target takeover rates using only training rollouts and initial takeover rates (revised §3.3).

On DAPO, segment averaging improved accuracy, and divergence-guided timing accelerated early learning. With identical BC on teacher continuations, the segment trigger outperformed the token-spike control at update 120 (submission Table 1). In an added seed-1 run, DTDD beat random takeover at update 40 (19.65% vs. 10.82%), but they were close by update 60 (revised §4.3; Appendix Table 13). We did not measure per-trigger precision/recall due to extra compute.

Thank you again for your careful review and for noting the value of retaining the student’s prefix and our boundary controls. In our rebuttal, we directly addressed your main soundness concerns with new experiments and matched comparisons:

  • Multi-seed & matched-update reliability (Appendix B.4.1): We added independent training runs across DAPO math (2 seeds), agentic code (2 seeds), and ALFWorld LoRA (3 seeds), showing that DTDD achieves higher mean macro scores (+6.29, +3.86, and +1.45 pp) and lower across-seed SDs than OPD at matched updates.
  • Controlled comparison against [-1, 1] clipped OPD (§4.3, Table 17): A new clipped-OPD run on the same base student, teacher, and seed confirms that clipping alone cannot reproduce DTDD’s early learning gain (at update 80: 21.34% vs. 13.40% for clipped OPD; reaching >20% at update 60 in 15.71h vs. update 200 in 20.21h on the same 4×L40S hardware), while DTDD also uses 55% fewer teacher tokens than offline SFT→OPD (43M vs. 95M).
  • Threshold tuning & exploration-floor ablation (§3.3, Appendix C.3): We clarified that thresholds are set solely from initial training rollouts without using evaluation data, and verified on 18K that disabling the 5% exploration floor leaves peak gradient norms (0.51 vs. 0.46) and accuracy (within 0.05 pp) virtually unchanged.

As the discussion period is nearing its end, may we ask whether these additions have addressed your main concerns? If so, we would greatly appreciate it if you could consider updating your Soundness and overall rating accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.