acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Evolving Distillation: Evolving Students through Teacher Adaptation

Abstract

On-Policy Distillation (OPD) has emerged as an effective paradigm for aligning and compressing large language models (LLMs) by matching the student’s on-policy data distribution. However, existing methods use a **fixed teacher and an evolving student**, limiting teacher–student co-evolution and risking teacher capability degradation. To address these challenges, we propose **O**n-**P**olicy **E**volving **D**istillation (**OPED**), a bidirectional distillation paradigm that **continuously evolves the teacher from student feedback**. OPED introduces two components: **Student-Driven Teacher Evolution (SDTE)**, which constructs preference pairs from student samples and updates the teacher via direct preference optimization, and **Global Privileged Information (GPI)**, a fixed teacher-only context that preserves the teacher’s core guidance capability during evolution. Extensive experiments on LLM distillation benchmarks and downstream tasks, including code generation and mathematical reasoning, show that OPED consistently improves student performance over static-teacher distillation, with gains of **0.93 and 3.04 points** in macro avg@12 accuracy for 1.7B and 8B models, respectively. Teacher-directed feedback also exceeds tuned direct student feedback by **0.76 points**, and the evolved teacher raises early one-token continuation utility by **0.70 points** on frozen student states. Further analysis shows that student feedback can **correct deficiencies in the teacher’s privileged information** as evolution progresses.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
On-Policy Distillation, Teacher Evolution, Student Feedback, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_MVHtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The submission proposes On-Policy Evolving Distillation (OPED), which extends OPD by periodically updating the teacher using feedback from the student. The method constructs preference pairs from student using verifiable correctness signals and use DPO to update the teacher.

Experiments on three math reasoning tasks with Qwen3-1.7B and Qwen3-8B demonstrate improvements over statis-teacher distillation and other baselines. Additional experiments examine the source and routing of feedback, teacher-updates schedules, and the usefulness of evolved teacher predictions on student-generated states.

Critical strengths
  1. The proposed method is intuitive and interesting. Updating the teacher using verified student trajectories is an interesting way to improve distillation performance.

  2. The empirical results shown in the manuscript show consistent improvements over the evaluation configurations. The authors conduct five training replicates for the main comparison and report confidence intervals. The ablation studies provide evidence supporting the claims.

Critical weaknesses
  1. Limited practical applicability and insufficient justification for the additional complexity of teacher evolution. Requiring the update of teachers is significantly more expensive and is inapplicable to common settings where the teacher is accessible only through inference or log-probability APIs. Also, the experiment setting is limited - teacher and student are from the same base family, which is considerably narrower than typical strong-to-weak distillation. It remains unclear whether the proposed approach is beneficial in more general setting given several potential confounding factors. In addition, the paper does not provide sufficient analysis of total training cost so it is difficult to determine whether the improvements justify the additional training complexity or whether comparable gains could be achieved by allocating the extra computation to simpler approaches.

  2. Limited reproducibility and insufficient validation of the empirical claims. The paper's contributions are primarily empirical but it is not clear how valid the results are without easy to execute artifacts for reproducibility. This is a significant concern because OPED involves a relatively complex training procedure with alternating student and teacher optimization, multiple LoRA adapters, verifier-based preference construction, periodic feedback collection, and changing teacher supervision.

Questions for authors
  1. What is the additional training cost introduced by teacher evolution, including teacher backward passes, optimizer states, preference-data generation, and DPO updates? Can the authors report end-to-end training GPU-hours and compare OPED with baselines under matched total compute budgets?

  2. Teacher stability and capability preservation. How does the teacher's performance evolve throughout training? Does teacher evolution lead to degradation on tasks outside the student feedback distribution?

  3. The role of GPI. What specifically makes fixed GPI an effective mechanism for preserving reliable teacher guidance during evolution? Does it measurably reduce teacher drift or prevent performance degradation? The current experiments demonstrate some benefits of procedural prompting but do not appear to establish the claimed stabilizing mechanism.

  4. Reproducibility. Given the complexity of the implementation and the centrality of the empirical gains, this would materially improve confidence in the paper.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer MVHt (Part 1: Compute-Matched Baselines [Q1])

We are deeply grateful to Reviewer MVHt for the generous and insightful review! We truly appreciate your recognition of OPED as an intuitive paradigm, along with our presentation clarity, five-replicate rigor, and ablations. Your sharp questions on compute efficiency and teacher stability hit the exact core of strengthening our paper; below we address Q1 with new profiling and experiments.

Q1: Training Cost and Compute-Matched Baselines

1. Runtime, GPU-Hours, and Memory Overhead

On 2 × RTX PRO 6000 GPUs, end-to-end OPED for Qwen3-1.7B (90 steps; teacher updated every 10 steps from 10 to 80) takes 2h 42m 52s (5.43 GPU-h), vs. ≈15m (≈0.50 GPU-h) for 1× static OPCD:

StepsStudent OPDSampling (750×8)Teacher DPOSubtotal
1–203m 05s14m 06s6m 50s24m 01s
21–403m 28s49m 18s19m 45s1h 12m 31s
41–603m 34s17m 20s9m 57s30m 51s
61–803m 25s20m 32s9m 35s33m 32s
81–901m 57s——1m 57s
Total (GPU-h)15m 29s (0.52h, 9.5%)1h 41m 16s (3.38h, 62.2%)46m 07s (1.54h, 28.3%)2h 42m 52s (5.43h)

Both models are rank-64 LoRA adapters on one frozen backbone, and τref\tau_{\text{ref}} reuses the frozen base model with GPI. Since student OPD and teacher DPO alternate sequentially, backward passes and optimizer states apply only to rank-64 LoRA weights (negligible peak-memory overhead). Extra runtime comes mainly from student sampling (62.2%) rather than teacher DPO (28.3%).

2. Matched-Compute Baselines (5.0–5.4 GPU-Hours)

On the same hardware, we compare OPED against 10-epoch static OPCD (≈2h 30m, ≈5.0 GPU-h) and GRPO (≈2h 30m, ≈5.0 GPU-h; reporting even the best epoch of the best among 3 new seeds):

Method (Qwen3-1.7B)ComputeMetricAIME24AIME25HMMT25Mean
OPCD (1×, Table 2)≈0.5 GPU-havg@1253.640.927.740.8
OPCD (10 epochs, Epoch-10)≈5.0 GPU-havg@643.331.719.431.5
OPCD (10 epochs, Best-epoch)≈5.0 GPU-havg@653.940.628.340.9
GRPO (Table 3)≈5.0 GPU-havg@1247.835.022.835.2
GRPO (Best of 3 new seeds)≈5.0 GPU-havg@650.636.724.437.2
OPED (Ours, Table 2)5.4 GPU-havg@1255.143.530.242.9

Training static OPCD 10× longer saturates (40.9% best-epoch) and overfits (31.5% at epoch 10), while even the best-seed, best-epoch GRPO (37.2%) trails OPED (42.9%) by 5.7 points.

AuthorsOpenReview ↗

Author Response to Reviewer MVHt (Part 2: Teacher Stability, GPI Stabilization & Reproducibility [Q2–Q4])

Following our Part 1 response on compute-matched baselines (Q1), we now address Q2–Q4 with new multi-seed teacher evaluations and a reproducibility plan. Thank you again for these constructive questions!

Q2: Teacher Stability Across Training

Tracking the evolved Qwen3-1.7B teacher (conditioned on GPI) across 3 training seeds on AIME2024 (3 seeds × avg@3) shows a brief early dip followed by strong recovery and net improvement:

Evolved Teacher CheckpointInitial Base + GPI (Table 1)Step 10Step 30Step 50Step 70
AIME2024 (3 seeds × avg@3)45.0% (avg@12)44.8%43.0%54.4%53.7%

The minor dip at step 30 (43.0%) coincides with the transient surge in student rollout length during exploration (steps 21–40 in Q1, where 750×8 sampling rises to 49m 18s), briefly perturbing teacher generation during DPO. Once student rollouts stabilize, teacher accuracy rebounds sharply to 54.4% (step 50) and 53.7% (step 70)—surpassing initial Base + GPI (45.0%) by +8.7 to +9.4 pp and Base (49.4%, Table 1) by +4.3 to +5.0 pp.


Q3: Role of Fixed GPI in Stabilizing Teacher Evolution

To directly quantify how fixed GPI acts on the evolving teacher, we evaluated the evolved teacher weights across 3 training seeds on AIME2024 (3 seeds × avg@3) with vs. without GPI, measuring both accuracy and token-length drift:

Teacher Checkpoint (3 Seeds × avg@3)Step 10Step 30Step 50Step 70
Accuracy with GPI44.8%43.0%54.4%53.7%
Accuracy without GPI50.7%51.5%51.1%52.2%
Mean Tokens with GPI13,14417,88721,07323,050
Mean Tokens without GPI15,67218,40624,69925,727
  • Curbing reasoning-length drift while lifting late-stage accuracy: As iterative DPO adapts the teacher on student trajectories, generation length drifts upward—inflating to 24,699–25,727 tokens (+64%) without GPI. By instructing the model to identify targets/constraints first, plan structure over unstructured trial-and-error, and verify, fixed GPI consistently suppresses internal thinking-token overhead across all checkpoints (saving 2,528 tokens at step 10, 3,626 tokens [-14.7%] at step 50, and 2,677 tokens [-10.4%] at step 70, while final-answer length stays nearly invariant). Crucially, although GPI initially lowers unadapted accuracy (44.8% vs. 50.7% at step 10, consistent with Table 1), parameter evolution learns to internalize and leverage this stationary procedural rubric, overtaking unconditioned evaluation in later stages (54.4% / 53.7% vs. 51.1% / 52.2%). This aligns with Table 7, where evolved-teacher utility concentrates at early planning positions (+0.7 pp).
  • Preventing prolonged student dips & why GPI stays fixed: Removing GPI from step 10 prolongs early student validation decline through step 30 and lowers final test accuracy (40.9% vs. 42.2%, Tables 8–9); meanwhile, random fixed tokens fail to anchor evolution (+0.4 pp, Table 6) and dynamically rewriting privileged prompts reaches only 38.8% vs. OPED's 42.9% (Table 10), confirming that keeping procedural rules stationary provides a reliable anchor during evolution.

Q4: Reproducibility

We fully agree that, given OPED's alternating student–teacher optimization, LoRA adapters, verifier-based preference construction, and periodic feedback collection, releasing complete artifacts is essential for confidence in the empirical claims. To this end, we will release a full reproduction package upon acceptance (for the camera-ready version), including:

  • Training & evaluation code: end-to-end OPED / OPCD pipelines, student reverse-KL distillation, teacher DPO, 750×8 feedback sampling, verifier-based preference-pair construction (mixed-success filtering and the 4-pair-per-problem cap), and evaluation scripts for AIME2024 / AIME2025 / HMMT25.
  • Configs & protocol files: all hyperparameters from Appendix A (learning rates, LoRA rank/scale, β\beta, KL cap, batch sizes, sampling temperatures, length limits, and enable_thinking=False settings), plus the exact train / feedback / model-selection / diagnostic pool splits.
  • Scripts for the new rebuttal analyses: wall-clock / GPU-hour profiling, matched-compute OPCD and GRPO baselines, Static OPCD + Student RFT, teacher AIME trajectories with/without GPI, and token-length drift measurements.

We will also release a concise README with one-command launch scripts so that the full OPED pipeline can be reproduced end-to-end on 2×80GB-class GPUs. We believe this package fully addresses the reproducibility concern and will further strengthen confidence in our empirical results. Thank you again for your valuable feedback!

AuthorsOpenReview ↗

Follow-up: Reviewer MVHt

Dear Reviewer MVHt,

I hope this message finds you well.

As the discussion phase is nearing its end, we would like to kindly check whether our previous responses have addressed your concerns. We fully understand that this is quite a busy period, and we sincerely appreciate the time and effort you have devoted to reviewing our work.

If there are any remaining questions, we would be more than happy to clarify them to the best of our ability.

Best regards,

The Authors

Reviewer_A6woOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes On-Policy Evolving Distillation, which extends on-policy context distillation by also updating the teacher during training. A same-size teacher LoRA, conditioned on a fixed teacher-only prompt, supervises the student with token-level reverse KL. Periodically, the teacher is updated with DPO on verified correct/incorrect pairs sampled from the current student. On AIME24/25 and HMMT25 with Qwen3-1.7B and Qwen3-8B, OPED improves macro avg@12 over a static-teacher OPCD baseline by 2.1 and 5.3 points. Ablations cover feedback routing, feedback source, GPI content, and one-token teacher utility on frozen student states.

Critical strengths
  • I really like the core conceptual point of this paper: a teacher's teaching value is relational (depends on the teacher, the student, and the states the student actually visits) and doesn't necessarily line up with the teacher's standalone accuracy. Showing that a procedural GPI prompt hurts the teacher's own accuracy (35.9 -> 33.3) while making it a much better OPCD teacher for the student is a very cool demonstration of this.
  • The proposed OPED method is clean and simple: periodically run DPO on the teacher using verifier-labeled pairs sampled purely from the current student, while keeping the teacher's context fixed.
  • Main results are run with 5 paired seeds and report bootstrap CIs across both seeds and problems.
  • The diagnostic experiments are well thought out—covering feedback routing, feedback source, shuffled-label controls, off-topic/random-token prompt controls, and frozen-state single-token utility.
  • Implementation details are clearly documented.
Critical weaknesses
  • Unmatched outcome supervision. OPED gives the teacher ground-truth verifier labels on 750 feedback problems, while baseline OPCD receives zero correctness rewards. Gains may simply reflect outcome supervision rather than teacher adaptation. A fair baseline must give static-teacher OPCD the same verified feedback.
  • Weak baselines and unmatched compute. GRPO underperforms the 1.7B base model, SFT degrades both scales, and alternating student DPO drops well below plain OPCD. Total compute (including feedback sampling and teacher DPO) is not reported.
  • Narrow evaluation scope. Only 90 test problems total, one domain, one model family, and a single fixed checkpoint (step 90).
  • Train/eval mode mismatch. Feedback rollouts use "enable_thinking=False" and are capped at 8K, while evaluation uses 39K tokens (thinking mode). This discrepancy is neither justified nor ablated.
Questions for authors
  1. How does OPED compare against static-teacher OPCD + student RFT/GRPO trained on the exact same verified rollouts from the 750-problem feedback pool?
  2. What is the total compute cost (including rollout generation and teacher DPO) for each method?
  3. How were the update schedule and step-90 checkpoint selected?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer A6wo

We cannot thank Reviewer A6wo enough for such a generous, perceptive review! We are thrilled that our relational view of teaching value resonated with you, and deeply appreciate your praise for OPED's clean design and diagnostics. Your brilliant Q1 suggestion uncovered a weight-update conflict that greatly deepens our paper!

Q1 & W1: Outcome-Matched Static OPCD + Student RFT & Update Geometry

Giving static OPCD the identical verified feedback as OPED on 1.7B (750×8 rollouts at steps 10–80) via interleaved Student RFT yields 36.1% (AIME24/25/HMMT25: 46.7/40.0/21.7), mirroring Student DPO:

Feedback Routing (Qwen3-1.7B, Identical 750-Problem Verifier Pool)MetricMean
Static OPCD (no outcome feedback, Table 2)avg@1240.8
Static OPCD + Student DPO (Table 4)avg@1237.0
Static OPCD + Student RFT (New)avg@636.1
OPED: Teacher DPO + Student OPD (Ours, Table 2)avg@1242.9

SVD of stage increments ΔW\Delta W across 196 adapted matrices (steps 20–80, k=32k=32) explains this 6.8 pp gap:

Stage Increment (Steps 20–80, k=32)Left AngleRight AngleCosine
Same-round OPCD vs. RFT78.14°28.72°-0.148
Consecutive OPCD27.55°8.70°+0.930
Consecutive RFT78.86°19.69°+0.050

Beyond objective differences (reverse KL vs. verified CE), 99.7% of same-round OPCD/RFT updates have negative cosine (-0.148), and RFT rotates sharply across rounds (+0.050 vs. OPCD's +0.930; divergence lies mainly in left subspaces as right angles narrow at k=64k=64). Updating student θ\theta causes static-teacher KL steps to overwrite RFT/DPO ((1−η)h(1-\eta)^h, App. B), whereas OPED updates teacher ϕ\phi into a persistent target ([1−(1−η)h][1-(1-\eta)^h]).


Q2 & W2: Compute Cost and Matched-Compute Baselines

On 2 × RTX PRO 6000, 1.7B OPED takes 2h 42m 52s (5.43 GPU-h): Student OPD 15m 29s (0.52h), 750×8 sampling 1h 41m 16s (3.38h), Teacher DPO 46m 07s (1.54h), vs. ≈0.5h (1× OPCD) and ≈5.0h (GRPO). At matched ≈5 GPU-h, 10-epoch OPCD saturates/overfits and 3-seed best GRPO beats Base (37.2% vs. 35.9%; SFT drops from off-policy shift) yet trails OPED by 5.7 pp:

Method (Qwen3-1.7B)GPU-hMetricMean
Base / 1× OPCD (Table 2–3)— / ≈0.5avg@1235.9 / 40.8
10× OPCD (Epoch-10 / Best: 53.9, 40.6, 28.3)≈5.0avg@631.5 / 40.9
GRPO (3-seed Best: 50.6, 36.7, 24.4)≈5.0avg@637.2
OPED (Ours, Table 2)5.4avg@1242.9

Q3, W3 & W4: Schedule, Protocol, and Scope

  • Schedule & Step-90 (Q3): We used uniform 10-step blocks (updates at steps 10–80; vs. 40.2% at step 50). Since 5-seed 39K evaluation is costly and validation had already plateaued once the post-step-80 block finished at step 90 (67.4%→68.1%→68.2% over steps 70–90, Table 9), we evaluated early and locked step 90 uniformly across all runs without cherry-picking.
  • Train/eval mode (W4): Inherited from OPSD (Zhao et al., 2026): enable_thinking=False (8K cap) for tractable sampling and 39K evaluation, shared identically across OPSD, OPCD, and OPED.
  • Scope (W3): 90 problems × 12 rollouts × 5 seeds (5,400 rollouts/method) give tight paired bootstrap CIs. Thank you again for elevating our work!
AuthorsOpenReview ↗

Follow-up: Reviewer A6wo

Dear Reviewer A6wo,

I hope this message finds you well.

As the discussion phase is nearing its end, we would like to kindly check whether our previous responses have addressed your concerns. We fully understand that this is quite a busy period, and we sincerely appreciate the time and effort you have devoted to reviewing our work.

If there are any remaining questions, we would be more than happy to clarify them to the best of our ability.

Best regards,

The Authors

Reviewer_ANYDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OPED, which alternates student distillation with DPO of a separate teacher using verified response pairs from the current student. The teacher retains fixed privileged context throughout training. Experiments with Qwen3 family model on three math tasks report improvements over static-teacher OPCD, with ablations examining the effects of teacher adaptation and context.

Critical strengths
  • The idea of updating a separate teacher with verified preference pairs from the current student is an interesting extension of OPD.
  • The research framework is clear, and the paper's structure is logical.
Critical weaknesses
  • While the idea is interesting, I am concerned about its practical application. For example, when the teacher is large, updating it during KD requires additional training resources, and the usefulness of the adapted teacher beyond the current distillation process is unclear. At 1.7B, the student accuracy improves from 40.8% with OPCD to 42.9% with OPED, but it is unclear whether this gain justifies the extra cost. The paper also provides no experiments quantifying GPU memory, training time, or overall computational overhead.
  • The experiments are limited to Qwen3 family on three math benchmarks. The main OPED experiments use same-sized teachers and students, so scalability to larger models and transfer to smaller students is unclear. Although appendix provides a local illustration of target reuse, it only offers limited theoretical support for the conditions under which student-driven teacher updates improve learning.
Questions for authors
  • What practical application setting is OPED intended for, particularly when the teacher is substantially larger than the student?
  • What are the additional GPU memory and training-time costs of teacher evolution, and do the gains over OPCD remain under matched total compute budgets?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer ANYD

We sincerely thank Reviewer ANYD for the thoughtful and practical review! We truly appreciate your recognition of student-driven teacher evolution as an interesting extension of OPD and your kind remarks on the clarity and logical structure of our paper. Your two questions pinpoint the exact practical boundaries and resource trade-offs of OPED; below we address Q1 and Q2 directly with new empirical measurements.

Q1: Practical Application Setting & When the Teacher Is Much Larger Than the Student

  • Why OPED is not intended for updating a much larger teacher: When the teacher is substantially larger than the student, preference pairs sampled from the smaller student cannot provide effective DPO supervision and may even degrade the larger teacher. Across the 750 × 8 repeated feedback rollouts, even at the end of training the 1.7B student reaches only ≈68% avg@8 accuracy (68.2%, Table 9), whereas our new measurement on the exact same 750 × 8 feedback rollouts shows that untrained Qwen3-8B already achieves 71.8% avg@8—surpassing the fully trained 1.7B student before any adaptation.
  • Intended practical setting (same-size evolution & context internalization): Accordingly, following OPCD (Ye et al., 2026) and OPSD (Zhao et al., 2026), OPED is designed for same-size post-training: (i) internalizing task-level rules (GPI) into a zero-prompt student with zero extra deployment cost, and (ii) self-evolving the largest affordable model at a target scale when no stronger teacher is available (e.g., improving Qwen3-8B from 62.0% with OPCD to 67.2% with OPED, +5.3 pp, Table 2). Moreover, for training a compact 1.7B student, same-size 1.7B OPED (42.9%) already outperforms standard OPD from a static 8B teacher (39.5%, Table 5) without hosting or updating a larger teacher.

Q2: Additional GPU Memory, Training Time, and Matched-Compute Comparison

1. GPU Memory and Wall-Clock Runtime (Qwen3-1.7B on 2 × RTX PRO 6000 GPUs)

  • GPU memory overhead: Both student and teacher are rank-64 LoRA adapters sharing one frozen backbone, and τref\tau_{\text{ref}} reuses the frozen base model with GPI without duplicating backbone weights. Because Student OPD and Teacher DPO run sequentially rather than concurrently, backward passes and Adam optimizer states are maintained only for the rank-64 LoRA adapter parameters, introducing minimal memory overhead over static-teacher distillation.
  • Training time: End-to-end 1.7B OPED (90 steps; teacher updated every 10 steps from 10 to 80) takes 2h 42m 52s (5.43 GPU-h), vs. ≈15m (≈0.50 GPU-h) for 1× static OPCD:
StepsStudent OPDSampling (750×8)Teacher DPOSubtotal
1–203m 05s14m 06s6m 50s24m 01s
21–403m 28s49m 18s19m 45s1h 12m 31s
41–603m 34s17m 20s9m 57s30m 51s
61–803m 25s20m 32s9m 35s33m 32s
81–901m 57s——1m 57s
Total (GPU-h)15m 29s (0.52h, 9.5%)1h 41m 16s (3.38h, 62.2%)46m 07s (1.54h, 28.3%)2h 42m 52s (5.43h)

2. Gains Remain Under Matched Total Compute Budgets (5.0–5.4 GPU-Hours)

On the same hardware, we trained static OPCD for 10 epochs (≈2h 30m, ≈5.0 GPU-h) and also compared against compute-matched GRPO (≈2h 30m, ≈5.0 GPU-h; best of 3 new seeds):

Method (Qwen3-1.7B)ComputeMetricAIME24AIME25HMMT25Mean
Static OPCD (1×, Table 2)≈0.5 GPU-havg@1253.640.927.740.8
Static OPCD (10 epochs, Epoch-10)≈5.0 GPU-havg@643.331.719.431.5
Static OPCD (10 epochs, Best-epoch)≈5.0 GPU-havg@653.940.628.340.9
GRPO (Best of 3 new seeds)≈5.0 GPU-havg@650.636.724.437.2
OPED (Ours, Table 2)5.4 GPU-havg@1255.143.530.242.9

Under matched compute, 10-epoch static OPCD saturates at its best epoch (40.9%) and overfits by epoch 10 (31.5%), and the strongest GRPO run (37.2%) remains 5.7 pp behind OPED (42.9%)—confirming that OPED's gains over OPCD persist under matched total compute budgets. Thank you again for your valuable feedback!

Thanks for your responses, which partial addresses my concern. However, as shown in this response, the OPED is not suitable for the case of that the size of teacher is larger than that of student. In same-size post-training, OPED needs extra computional resources. As shown in table in your response 2, Static OPCD (1×) already has a good performance (40.8) under 0.5 GPU-h, and OPED improves it by 2.1 under 5.4 GPU-h. Using huge computational overhead for a minor gain let me think this method is limited. Combining the other weaknesses I mentioned before, I keep my score.

Reviewer_rM4yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OPED, which alternates student on-policy distillation with teacher adaptation. Correct and incorrect responses sampled from the current student form preference pairs for periodic teacher DPO updates. The teacher receives fixed privileged guidance, GPI, which is not directly provided to the student. Experiments with Qwen3-1.7B and Qwen3-8B cover three competition-mathematics benchmarks, different feedback sources and update routes, and one-token interventions on frozen student states. Across five training replicates, OPED improves macro-average accuracy over static-teacher OPCD by approximately 2.1 and 5.3 percentage points, respectively.

Critical strengths
  1. The distinction between standalone teacher accuracy and teaching utility is meaningful and empirically examined. The frozen-student interventions, including correctly labeled and shuffled-label DPO controls, provide more direct evidence of changes in local guidance quality than final student accuracy alone.

  2. The main improvements are supported by repeated experiments and targeted controls. Five paired seeds and simultaneous confidence intervals support gains at both model sizes. Comparisons using current-student, initial-student, and teacher-generated feedback also provide evidence for refreshing feedback as the student evolves.

Critical weaknesses
  1. Teacher evolution adds substantial verified supervision and computation. Each adaptation round samples eight responses for each of 750 feedback problems before teacher DPO. Static OPCD does not consume the same additional feedback, so the main comparison establishes the value of the complete procedure without establishing that teacher-directed use of these resources is preferable to direct student training.

  2. The mechanistic evidence is narrower than the downstream performance evidence. The significant frozen-state utility improvement occurs at early positions, with no clear improvement at late positions. It therefore does not yet explain the benefit across a complete reasoning trajectory.

Questions for authors
  1. What are the additional rollout tokens, verification calls, and teacher-training costs of OPED? How does it compare with direct student DPO or GRPO using the same feedback pool and a comparable total budget?

  2. Does the student-DPO control use identical preference pairs, optimization steps, and tuning effort? How can the benefit of teacher adaptation be separated from the effect of pulling direct student updates back toward a fixed teacher?

  3. Which student checkpoint and teacher update are used in the frozen-state intervention, and how were early and late positions prespecified? Does the static-versus-evolving gap persist when GPI is omitted from the beginning?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer rM4y

We are truly grateful to the reviewer for your thoughtful, fair, and encouraging evaluation, and especially for appreciating the core distinction between standalone teacher accuracy and teaching utility, the frozen-state intervention design, and our repeated paired-seed controls. Your questions are remarkably insightful and directly help us sharpen the resource accounting and experimental reporting. Below we address each point in detail.

Q1: Rollout Tokens, Verification Calls, Teacher-Training Costs, and Matched-Budget Comparisons

On Qwen3-1.7B (2 × RTX PRO 6000 GPUs), OPED runs 90 student steps with 8 evolution boundaries (steps 10–80). Relative to 1× static OPCD (≈15m, ≈0.50 GPU-h), the additional costs are:

  • Rollout tokens & verification calls: 750 × 8 = 6,000 responses per boundary (48,000 responses and 48,000 verification calls total). Mean completion length is 1,013.67 output tokens (vs. 8,192 cap), totaling ≈6.08M rollout tokens/round and ≈48.66M rollout tokens across 8 rounds.
  • Wall-clock & GPU-hours: Total 2h 42m 52s (5.43 GPU-h)—student OPD 15m 29s (0.52 GPU-h, 9.5%), feedback sampling 1h 41m 16s (3.38 GPU-h, 62.2%), and teacher DPO + reference scoring 46m 07s (1.54 GPU-h, 28.3%).

Comparing OPED against direct student training on the same 750-problem feedback pool (Student DPO / new Student RFT) and 10× extended static OPCD under a comparable ≈5.0–5.4 GPU-h budget:

Method (Qwen3-1.7B)Extra Feedback / BudgetEvalMacro Mean
Static OPCD (1×, Table 2)none / ≈0.5 GPU-havg@1240.8
Static OPCD + Student DPO (Table 4)same 750×8 pool / ≈5.4 GPU-havg@1237.0
Static OPCD + Student RFT (New)same 750×8 pool / ≈5.4 GPU-havg@636.1
Static OPCD 10 epochs (Best / Final)none / ≈5.0 GPU-havg@640.9 / 31.5
OPED (Table 2)same 750×8 pool / 5.43 GPU-havg@1242.9

Neither routing the same feedback pool directly to the student (36.1%–37.0%) nor allocating ≈5.0 GPU-h to longer static OPCD (≤40.9%) matches OPED (42.9%).


Q2: Student-DPO Control Fairness

The student-DPO control in Table 4 uses the exact same DPO hyperparameters, optimization steps, and tuning effort as teacher DPO (lr = 10−610^{-6}, β=0.2\beta = 0.2, batch size 8, steps 10–80, with no separate tuning on either branch). Preference pairs are not shared across branches; each branch runs the identical on-policy 750×8 sampling and verifier-based pair-construction protocol on its own current student.


Q3: Frozen-State Setup and Scope of Early-Position Evidence

  • Checkpoints & positions (Table 7): The frozen student is the step-90 student checkpoint. Teachers compared are the initial teacher (before DPO) and the step-80 (final) evolved teachers trained from initialization over the same 8 rounds—comparing correct-label DPO against a matched shuffled-label DPO run in which chosen and rejected responses are swapped on ≈15% of preference pairs across rounds. For each full trajectory sampled from the frozen student under the evaluation decoding setting (averaging ≈22,189 tokens per response), the prespecified early position is token 512 from the start and the late position is 512 tokens before the end.
  • Scope of early-position evidence: We agree this one-token probe is narrower than full-trajectory dynamics and will state this explicitly. Token 512 lies in early problem planning where a single-token shift can redirect the solution path, whereas 512 tokens before completion is near wrap-up where the trajectory is largely committed; in OPED, improved early guidance compounds across repeated on-policy distillation steps.

We will incorporate these exact cost numbers, frozen-state definitions, and scoped mechanistic remarks into the revision. Thank you again for your encouraging review and insightful feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.