acceptodds
Under review as a conference paper at ICLR 2027

Curriculum-OPD: Efficient Multi-Teacher Multi-Domain On-Policy Distillation

Abstract

On-policy distillation (OPD) transfers knowledge through token-level teacher supervision on student-generated responses. In practical applications, a student model often needs to handle tasks such as mathematical reasoning and code generation, and each task may have several potential teachers. Training in this setting requires repeated response generation and teacher evaluation, incurring substantial computational cost. To improve distillation efficiency, we examine how teacher feedback and task gains evolve during training and find that continued distribution alignment does not necessarily yield continued task improvement. We distinguish contextual and background feedback by comparing teacher–student log-probability differences with and without the input question. Background feedback gradually becomes dominant as task gains diminish, suggesting that contextual feedback may provide more direct guidance for learning task-specific knowledge. Based on this analysis, we propose Curriculum-OPD, an efficient framework for multi-teacher multi-domain OPD. The framework selects training examples using contextual feedback on short student responses. On-Policy Hybrid Distillation (OPHD) assigns external teacher supervision or reference-conditioned self-distillation to each example. Domain-balanced update projection further addresses differences in gradient scale and conflicting update directions during joint training. Experiments on seven benchmarks across mathematics, code, and medicine demonstrate that Curriculum-OPD outperforms standard OPD in all three domains. Learning-curve comparisons further show that Curriculum-OPD reaches comparable performance in fewer training updates. Code is available at: https://anonymous.4open.science/r/Curriculum-OPD-574D/.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
curriculum learning, on-policy distillation, multi-teacher distillation, multi-domain learning, self-distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SWunOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Curriculum-OPD, a framework for efficient multi-teacher, multi-domain OPD. Starting with the observation that the relationship between teacher-student alignment and downstream performance, the authors propose Curriculum-OPD which consists of three part: (1) sample selection based on contextual feedback estimated from short student responses, (2) question-level supervision selection among external teachers and reference-conditioned self-distillation, and (3) domain-balanced update projection to coordinate joint training. Experiments on seven benchmarks coving different domains, and the results show improvements over standard OPD and ExOPD.

Critical strengths

1: The paper addresses a meaningful problem in LLM post-training. Improving the efficiency of multi-teacher OPD is an important and timely problem to explore.

2: Direct motivation and interesting method design. The distinction between feedback with and without the input question provides an interesting perspective on teacher-student disagreement. The proposed framework connects sample selection, supervision assignment, and multi-domain optimization in a singe training pipeline, which is a great attempt from my perspective.

Critical weaknesses

1: The paper uses a 16 token preview for sample selection. However, for complex tasks, such as mathematical reasoning, the main difficulties may arise much later in the response. It is unclear whether such a short preview can reliably reflect the learning value of a question. I would encourage the authors to compare different preview lengths, such as 8, 16, 32, 64, and longer sequences, and investigate how the preview length affects both sample selection quality and downstream performance.

2: Computational efficiency claim remains a major concern for me. The paper primarily demonstrates efficiency in terms of training updates rather than total computational cost. According to Appendix C, the proposed method screens 56,608 mathematical questions and 25,276 code questions, each requiring student generation and four log-probability scoring views. In addition, OPHD requires generating a full initial-student response for each selected question and scoring it with multiple eligible supervision sources before training. Domain-balanced update projection also introduces additional gradient computations during training. These costs are particularly important given that the main experiments involve only 7 or 14 training updates. Although Figure 3 shows faster improvement in terms of updates, it remains unclear whether the savings outweigh the additional preprocessing and optimization costs. I would encourage the authors to report end-to-end GPU hours, including all preprocessing stages, and compare performance under matched computational budgets.

3: The interpretation of the early training dynamics in Figure 1 lacks detailed analysis. Figure 1(b) shows substantial changes in AIME24 accuracy within the first few training updates. However, Appendix D reports considerable response truncation during this period. For the 4B specialist, 59 and 35 out of 480 responses are truncated at updates 2 and 3, respectively. For the 30B teacher, the numbers are 74, 94, and 130 at updates 2, 3, and 4. These observations raise the question of whether changes in response length and truncation contribute to the observed performance fluctuations, rather than the proposed feedback dynamics alone. Notably, the accuracy decline of the 4B specialist at update 4 occurs without truncated responses, so truncation alone cannot explain all the fluctuations. See problem 3 for more details.

Questions for authors

1: Could you add a comparison with different token preview length, and provide results on how the preview length affect the performance?

2: Could you report the end-to-end GPU hours?

3: Could the authors provide a more detailed analysis of Figure 1(b), particularly why AIME24 accuracy increases from around 20% to 70% within only two training updates? According to the appendix, both training and evaluation use non-thinking mode. This raises the question of whether the rapid improvement comes partly from changes in the student's response length or its ability to generate longer reasoning steps, rather than solely from acquiring new reasoning capabilities. Moreover, Appendix D reports substantial response truncation during the early updates, suggesting that response length may also contribute to the observed performance fluctuations. It would be helpful to report the average response length, truncation rate, and accuracy across training updates to better understand these dynamics. Additionally, Figure 1 uses a 32,768-token generation limit, whereas the main experiments use 16,384 tokens. Could the authors clarify whether this difference affects the reported results? The reason I raise this is that both Qwen3 4B and 30B Instruct are models that produces long reasoning chains even in non-thinking mode, considering the length exploration recorded in the appendix, I do think it worthing taking this into consideration.

Reviewer_8WJpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Curriculum-OPD, an efficiency-oriented framework for multi-teacher, multi-domain OPD. The framework introduces a diagnostic metric that separates contextual teacher feedback from background feedback using short student previews to guide sample selection. It then utilizes On-Policy Hybrid Distillation (OPHD) with a "Gap" criterion to route per-sample supervision between external teachers and reference-conditioned self-distillation. Finally, a domain-balanced update projection is applied to mitigate cross-domain interference, with experiments demonstrating performance and efficiency gains across math, code, and medical benchmarks compared to standard OPD baselines.

Critical strengths
  • The separation of contextual and background feedback by scoring the same student tokens with and without the question is an actionable diagnostic that avoids costly long rollouts.

  • OPHD provides a clean, Gap-based heuristic that effectively unifies external teachers and self-distillation into a single per-sample decision on a shared baseline.

Critical weaknesses
  • The experimental evaluation compares the method against standard OPD, ExOPD, and OPSD, but completely omits critical, concurrent SOTA baselines in distillation curricula and projection-based multi-objective OPD (e.g., [1,2]). This makes it impossible to determine if the proposed method actually advances the current state-of-the-art or merely offers an alternative to older baselines.

  • While the paper's core premise is training efficiency, the results omit matched wall-clock time or concrete compute accounting (e.g., GPU-hours, teacher forward passes). Additionally, the reliance on single-seed runs without reporting statistical uncertainty makes the modest performance margins difficult to trust.

[1] Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation.

[2] ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation

Questions for authors
  • The medical benchmark evaluation relies exclusively on a local Qwen3-32B judge. Did you validate these results with alternative LLM judges or human spot-checks to ensure the chosen judge does not systematically bias the comparisons between the different distillation methods?

  • Given that efficiency is a core claim of the paper, can you provide concrete compute accounting—such as wall-clock time, GPU-hours, and the exact number of teacher forward passes—to back up the update-efficiency learning curves?

  • Most results appear to use a single nominal seed. Can you report results across multiple seeds with confidence intervals to prove the statistical significance of the reported gains, particularly in the single-domain medical evaluation where margins are modest?

  • How robust is the background-feedback subtraction to prompt-format choices (e.g., different chat templates or the presence/absence of a system prompt), and does the r−br-b contrast remain predictive of task gains under such variations?

  • Could you clarify how often OPHD chooses the self-distillation branch versus external teachers in each domain, and explicitly define what constitutes "verified reference information s" for the math and code domains?

Reviewer_Xia1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Curriculum-OPD, a framework intended to make multi-teacher, multi-domain on-policy distillation more efficient. It separates teacher–student log-probability differences into contextual feedback (with the question) and background feedback (with an empty question), and argues that background feedback dominates as task gains plateau. Questions are chosen using a clipped, sensitivity-weighted RMS on 16-token previews from the initial student. Each retained question is assigned, once and before training, to a single supervision source. The assignment is the source with the highest mean log-probability Gap on one cached student response. The domain gradients are normalized and averaged, and the Adam step is then projected (GEM-style) so that no domain–supervisor group’s linearized loss increases. Achieved performance gains for both single and joint runs. Learning curves claim fewer updates to reach comparable AIME accuracy.

Critical strengths
  1. The with/without-question diagnostic is an interesting lens on OPD. Separating question-dependent from generic teacher feedback is a reasonable way to ask why alignment keeps improving while task accuracy plateaus.

  2. The teacher-assignment control is well designed and the work is reproducible.

  3. The Random baseline keeps the same teacher-usage counts as Gap-based selection, and the pass@k coverage analysis separates consistency gains from broader coverage.

Critical weaknesses
  1. I feel the method is an incremental combination of three known components such as IFD-style with/without-question scoring for sample selection, per-sample teacher selection plus the existing ExOPD and OPSD losses for OPHD, and GEM/PCGrad-style gradient projection for the domain updates.

  2. The efficiency claim is not properly measured which I think is really important. No wall-clock time, GPU-hours or FLOPs are reported, and the paper explicitly disclaims computational speedup. Also, the method adds overhead: four teacher-forced passes over about 82k screening candidates, scoring by every source, and per-group gradients with a projection at each step.

  3. The authors discussed about update-efficiency curves in Fig. 3 but in my opinion are confounded. Curriculum-OPD at lr = 10⁻⁵ is compared against OPD/ExOPD at lr = 10⁻⁶. The conclusion describes the comparison as relative to published trajectories, which suggests the baseline curves are not matched reproductions.

  4. The motivating phenomenon lies outside the trained regime. Background dominance is shown around updates 10–25, but the main experiments stop at 7–14 updates, roughly one pass over the 1,792-question pool and largely within the rapid early-gain phase.

  5. I understand that gap-based teacher selection actually rewards agreement rather than the knowledge. The Gap scores basically the student’s own, possibly incorrect, response, so the argmax selects the teacher that most supports what the student already wrote. Also, I guess that It favors same-base teachers and is fixed before training, so I stuggle to understand the paper’s claims of adapting to the student’s state, may need to make it more clear to justify.

  6. The 16-token previews are too short to capture task-relevant knowledge. Both the screening score and the Fig. 1(c) diagnostics rely on them. The Section 3 evidence is also correlational, single-seed and noisy: HumanEval+ uses one response per question, and the 30B trajectory has 15–27% truncated responses at updates 2–4. Need to work on it, I think.

  7. The evaluation protocol need to be improved because the results are single-seed, the code score for Curriculum-OPD averages two runs while each baseline uses one, and Appendix B states that benchmark evaluations informed development with no separate validation set. So the 0.5–2 point margins are therefore unreliable.

  8. I would like to suggest to downstate the claims as they are overstated relative to the results reported in the paper, The key baselines are missing. Curriculum-OPD loses to ExOPD in medicine, and medicine is excluded from joint training.

  9. The paper also does not compare against the methods that seems important to report such as: MOPD, TrustMOPD, PG-OPD/IFD-style selection, PCGrad/CAGrad, task-arithmetic merging of the same-base specialists, or GRPO.

Questions for authors
  1. If possible, please report multi-seed means ± std for Tables 1–3, with every method averaged over the same number of runs.

  2. Were design choices or hyperparameters tuned on the test benchmarks? If so, please re-run on a held-out validation split.

  3. What are the measured wall-clock time and GPU-hours for screening, Gap scoring and projection, compared with plain OPD/ExOPD at equal final accuracy?

  4. Are the baseline curves in Fig. 3 from your own runs or from published papers? How do OPD and ExOPD perform at lr = 10⁻⁵ over the same early updates?

  5. Does Gap-based selection change if it is computed on a correct reference solution instead of the student’s own response?

  6. What is the effect of using r_t alone versus r_t − b_t, and of preview lengths longer than 16 tokens?

  7. Which question pool does each baseline in Tables 1–3 use? The ExOPD row in Table 3 is marked as using sample selection.

  8. Why is medicine excluded from joint training? Do the results hold for math + code + medicine? Better to specify or include medicine also in joint training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.