- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The submission proposes On-Policy Evolving Distillation (OPED), which extends OPD by periodically updating the teacher using feedback from the student. The method constructs preference pairs from student using verifiable correctness signals and use DPO to update the teacher.
Experiments on three math reasoning tasks with Qwen3-1.7B and Qwen3-8B demonstrate improvements over statis-teacher distillation and other baselines. Additional experiments examine the source and routing of feedback, teacher-updates schedules, and the usefulness of evolved teacher predictions on student-generated states.
Critical strengths
-
The proposed method is intuitive and interesting. Updating the teacher using verified student trajectories is an interesting way to improve distillation performance.
-
The empirical results shown in the manuscript show consistent improvements over the evaluation configurations. The authors conduct five training replicates for the main comparison and report confidence intervals. The ablation studies provide evidence supporting the claims.
Critical weaknesses
-
Limited practical applicability and insufficient justification for the additional complexity of teacher evolution. Requiring the update of teachers is significantly more expensive and is inapplicable to common settings where the teacher is accessible only through inference or log-probability APIs. Also, the experiment setting is limited - teacher and student are from the same base family, which is considerably narrower than typical strong-to-weak distillation. It remains unclear whether the proposed approach is beneficial in more general setting given several potential confounding factors. In addition, the paper does not provide sufficient analysis of total training cost so it is difficult to determine whether the improvements justify the additional training complexity or whether comparable gains could be achieved by allocating the extra computation to simpler approaches.
-
Limited reproducibility and insufficient validation of the empirical claims. The paper's contributions are primarily empirical but it is not clear how valid the results are without easy to execute artifacts for reproducibility. This is a significant concern because OPED involves a relatively complex training procedure with alternating student and teacher optimization, multiple LoRA adapters, verifier-based preference construction, periodic feedback collection, and changing teacher supervision.
Questions for authors
-
What is the additional training cost introduced by teacher evolution, including teacher backward passes, optimizer states, preference-data generation, and DPO updates? Can the authors report end-to-end training GPU-hours and compare OPED with baselines under matched total compute budgets?
-
Teacher stability and capability preservation. How does the teacher's performance evolve throughout training? Does teacher evolution lead to degradation on tasks outside the student feedback distribution?
-
The role of GPI. What specifically makes fixed GPI an effective mechanism for preserving reliable teacher guidance during evolution? Does it measurably reduce teacher drift or prevent performance degradation? The current experiments demonstrate some benefits of procedural prompting but do not appear to establish the claimed stabilizing mechanism.
-
Reproducibility. Given the complexity of the implementation and the centrality of the empirical gains, this would materially improve confidence in the paper.
3 comments
Author Response to Reviewer MVHt (Part 2: Teacher Stability, GPI Stabilization & Reproducibility [Q2–Q4])
Following our Part 1 response on compute-matched baselines (Q1), we now address Q2–Q4 with new multi-seed teacher evaluations and a reproducibility plan. Thank you again for these constructive questions!
Q2: Teacher Stability Across Training
Tracking the evolved Qwen3-1.7B teacher (conditioned on GPI) across 3 training seeds on AIME2024 (3 seeds × avg@3) shows a brief early dip followed by strong recovery and net improvement:
| Evolved Teacher Checkpoint | Initial Base + GPI (Table 1) | Step 10 | Step 30 | Step 50 | Step 70 |
|---|---|---|---|---|---|
| AIME2024 (3 seeds × avg@3) | 45.0% (avg@12) | 44.8% | 43.0% | 54.4% | 53.7% |
The minor dip at step 30 (43.0%) coincides with the transient surge in student rollout length during exploration (steps 21–40 in Q1, where 750×8 sampling rises to 49m 18s), briefly perturbing teacher generation during DPO. Once student rollouts stabilize, teacher accuracy rebounds sharply to 54.4% (step 50) and 53.7% (step 70)—surpassing initial Base + GPI (45.0%) by +8.7 to +9.4 pp and Base (49.4%, Table 1) by +4.3 to +5.0 pp.
Q3: Role of Fixed GPI in Stabilizing Teacher Evolution
To directly quantify how fixed GPI acts on the evolving teacher, we evaluated the evolved teacher weights across 3 training seeds on AIME2024 (3 seeds × avg@3) with vs. without GPI, measuring both accuracy and token-length drift:
| Teacher Checkpoint (3 Seeds × avg@3) | Step 10 | Step 30 | Step 50 | Step 70 |
|---|---|---|---|---|
| Accuracy with GPI | 44.8% | 43.0% | 54.4% | 53.7% |
| Accuracy without GPI | 50.7% | 51.5% | 51.1% | 52.2% |
| Mean Tokens with GPI | 13,144 | 17,887 | 21,073 | 23,050 |
| Mean Tokens without GPI | 15,672 | 18,406 | 24,699 | 25,727 |
- Curbing reasoning-length drift while lifting late-stage accuracy: As iterative DPO adapts the teacher on student trajectories, generation length drifts upward—inflating to 24,699–25,727 tokens (+64%) without GPI. By instructing the model to identify targets/constraints first, plan structure over unstructured trial-and-error, and verify, fixed GPI consistently suppresses internal thinking-token overhead across all checkpoints (saving 2,528 tokens at step 10, 3,626 tokens [-14.7%] at step 50, and 2,677 tokens [-10.4%] at step 70, while final-answer length stays nearly invariant). Crucially, although GPI initially lowers unadapted accuracy (44.8% vs. 50.7% at step 10, consistent with Table 1), parameter evolution learns to internalize and leverage this stationary procedural rubric, overtaking unconditioned evaluation in later stages (54.4% / 53.7% vs. 51.1% / 52.2%). This aligns with Table 7, where evolved-teacher utility concentrates at early planning positions (+0.7 pp).
- Preventing prolonged student dips & why GPI stays fixed: Removing GPI from step 10 prolongs early student validation decline through step 30 and lowers final test accuracy (40.9% vs. 42.2%, Tables 8–9); meanwhile, random fixed tokens fail to anchor evolution (+0.4 pp, Table 6) and dynamically rewriting privileged prompts reaches only 38.8% vs. OPED's 42.9% (Table 10), confirming that keeping procedural rules stationary provides a reliable anchor during evolution.
Q4: Reproducibility
We fully agree that, given OPED's alternating student–teacher optimization, LoRA adapters, verifier-based preference construction, and periodic feedback collection, releasing complete artifacts is essential for confidence in the empirical claims. To this end, we will release a full reproduction package upon acceptance (for the camera-ready version), including:
- Training & evaluation code: end-to-end OPED / OPCD pipelines, student reverse-KL distillation, teacher DPO, 750×8 feedback sampling, verifier-based preference-pair construction (mixed-success filtering and the 4-pair-per-problem cap), and evaluation scripts for AIME2024 / AIME2025 / HMMT25.
- Configs & protocol files: all hyperparameters from Appendix A (learning rates, LoRA rank/scale, , KL cap, batch sizes, sampling temperatures, length limits, and
enable_thinking=Falsesettings), plus the exact train / feedback / model-selection / diagnostic pool splits. - Scripts for the new rebuttal analyses: wall-clock / GPU-hour profiling, matched-compute OPCD and GRPO baselines, Static OPCD + Student RFT, teacher AIME trajectories with/without GPI, and token-length drift measurements.
We will also release a concise README with one-command launch scripts so that the full OPED pipeline can be reproduced end-to-end on 2×80GB-class GPUs. We believe this package fully addresses the reproducibility concern and will further strengthen confidence in our empirical results. Thank you again for your valuable feedback!
Follow-up: Reviewer MVHt
Dear Reviewer MVHt,
I hope this message finds you well.
As the discussion phase is nearing its end, we would like to kindly check whether our previous responses have addressed your concerns. We fully understand that this is quite a busy period, and we sincerely appreciate the time and effort you have devoted to reviewing our work.
If there are any remaining questions, we would be more than happy to clarify them to the best of our ability.
Best regards,
The Authors
Author Response to Reviewer MVHt (Part 1: Compute-Matched Baselines [Q1])
We are deeply grateful to Reviewer MVHt for the generous and insightful review! We truly appreciate your recognition of OPED as an intuitive paradigm, along with our presentation clarity, five-replicate rigor, and ablations. Your sharp questions on compute efficiency and teacher stability hit the exact core of strengthening our paper; below we address Q1 with new profiling and experiments.
Q1: Training Cost and Compute-Matched Baselines
1. Runtime, GPU-Hours, and Memory Overhead
On 2 × RTX PRO 6000 GPUs, end-to-end OPED for Qwen3-1.7B (90 steps; teacher updated every 10 steps from 10 to 80) takes 2h 42m 52s (5.43 GPU-h), vs. ≈15m (≈0.50 GPU-h) for 1× static OPCD:
Both models are rank-64 LoRA adapters on one frozen backbone, and τref reuses the frozen base model with GPI. Since student OPD and teacher DPO alternate sequentially, backward passes and optimizer states apply only to rank-64 LoRA weights (negligible peak-memory overhead). Extra runtime comes mainly from student sampling (62.2%) rather than teacher DPO (28.3%).
2. Matched-Compute Baselines (5.0–5.4 GPU-Hours)
On the same hardware, we compare OPED against 10-epoch static OPCD (≈2h 30m, ≈5.0 GPU-h) and GRPO (≈2h 30m, ≈5.0 GPU-h; reporting even the best epoch of the best among 3 new seeds):
Training static OPCD 10× longer saturates (40.9% best-epoch) and overfits (31.5% at epoch 10), while even the best-seed, best-epoch GRPO (37.2%) trails OPED (42.9%) by 5.7 points.