acceptodds
Withdrawn from ICLR 2027

D-OPD: Enhancing Inter-Teacher Discriminability in Multi-Task On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

For text-to-image (T2I) generation, existing reinforcement learning (RL) methods are constrained by reward hacking and gradient conflicts across different objectives, leading to poor performance in multi-task scenarios. Although the recent On-Policy Distillation (OPD) paradigm mitigates these challenges by constructing dense supervision signals, significant room for improvement remains. In particular, vanilla OPD merely enforces the alignment between the student model and the task-specific teacher without constraining its distributional divergence from the remaining ones. This prevents the model from disentangling the knowledge of irrelevant teachers when optimizing for a given task. To address this issue, we propose D-OPD, an efficient framework that reformulates the multi-teacher on-policy distillation as a task-expert classification problem. Concretely, we first derive a categorical distribution over teachers based on the student–teacher discrepancy, quantifying the likelihood of the student’s rollout states being attributed to each teacher. By improving the probability of the task-specific teacher via a cross-entropy objective, D-OPD pulls the student model closer to the corresponding task expert, while maintaining distributional separation from the rest. Benefiting from this, D-OPD significantly enhances the inter-teacher discriminability of the student’s rollouts, facilitating faster convergence and superior performance. In addition, D-OPD incorporates Dual-Teacher (DT) and Early-Window (EW) strategies to alleviate the computational overhead incurred by an increasing number of teachers, enhancing training efficiency without compromising the overall performance. Extensive experiments demonstrate that D-OPD consistently outperforms RL and OPD baselines across various settings, highlighting its significant promise for multi-task post-training.

Primary area:
generative models
Keywords:
on-policy distillation, text-to-image generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5SVpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets multi-task post-training of text-to-image flow models by using OPD from several task-specific teachers such as OCR, PickScore, and GenEval. The authors argue that original OPD leaves the student's rollout indistinguishable across teachers. The proposed InfoNCE-style cross-entropy loss that raises the probability of the task teacher and implicitly pushes the student away from the other teachers. Evaluation results of proposed D-OPD demonstrate faster reward convergence than original OPD and multi-reward GRPO.

Critical strengths
  • The method is simple yet efficient and easy to adopt on existing on-policy distillation for many application. The proposed DT and EW address the obvious cost of evaluating all teachers at every step.

  • The empirical improvement is significant. Convergence is much faster than vanilla OPD and MR-GRPO, consistently across 2, 3, and 5 teacher settings.

Critical weaknesses
  • The evaluation results are statistically unsupported. the high-variance and unstable nature of reinforcement learning, multi-seed run and significance test are required for evaluation. However, there are no multi-seed statistic, error bars, or confidence interval. Also test-split metrics are the same reward models used to train the teachers, so they cannot detect reward hacking. Multi-seed run and statistical report and qualitative results and human evaluation might address this concern.

  • DT and EW are motivated by computational cost, but there are no wall-clock time, NFE count, or memory figures are reported. Also convergence speed is measured per training iteration. This may not mean real efficiency of training cost.

Questions for authors
  • Please report results over multiple seeds with error bars, plus wall-clock time and NFEs per iteration for vanilla OPD, D-OPD w/o DT/EW, D-OPD w/o EW, and full D-OPD.
Reviewer_TBeJOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes D-OPD, which reformulates multi-task on-policy distillation as task-expert classification using probabilities derived from student–teacher KL divergences. It further introduces Dual-Teacher and Early-Window strategies to reduce teacher computation. Experiments with SD3.5-M report improved task performance and iteration-based convergence across multiple task settings.

Critical strengths

Relative teacher-aware distillation objective. The method incorporates the student’s relative distances to multiple teachers rather than matching only the designated teacher. This provides a concrete and implementable formulation for organizing supervision from multiple task specialists.

Critical weaknesses
  1. Insufficient validation of the central mechanism. Vanilla OPD already exhibits a teacher-probability gap in Figure 2, but the paper does not provide a clear criterion or justification for calling this gap “limited” and the student rollouts “indistinguishable across teachers.” Nor does it establish how this degree of separation causes performance loss or slower convergence. These gaps weaken the central motivation for D-OPD.
  2. Insufficient evidence for computational efficiency. The convergence plots measure progress by iterations without accounting for differences in teacher computation per iteration. Without runtime, GPU-hour, memory, and matched-performance cost comparisons, the actual savings from DT and EW remain unclear. This leaves a central claimed contribution insufficiently substantiated.
Questions for authors
  1. Why is the task-matched teacher considered the expert while the others are treated as “irrelevant”? Since other teachers may provide complementary knowledge, such as aesthetics and layout, could repelling them harm performance on prompts requiring multiple capabilities?
  2. Although the ablations show benefits from EW, what are the separate contributions of early CE and subsequent OPD? Is returning to OPD necessary, or do the gains mainly come from early CE? Do any experiments distinguish these effects?
  3. What do the nodes and arrows in Figure 3 represent? How are state transitions distinguished from teacher guidance? How exactly are student and teacher predictions at the same noisy state used to compute KL divergences, teacher probabilities, and the training loss?
  4. Some outputs in Figure 6 appear stylistically similar to those of the PickScore teacher. Does this similarity reflect an excessive bias toward the preference teacher? Does the method affect the stylistic diversity of generated images?
Reviewer_UPScOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes D-OPD for combining task-specific text-to-image teachers into one student. It uses student–teacher KL divergences to define a classification objective that favors the teacher assigned to the current task over the other teachers. To reduce computation, it samples one negative teacher and applies this objective only during early denoising steps. Experiments cover OCR, image preference, compositional generation, and a five-teacher setting with customization tasks.

Critical strengths
  1. The improvements are consistent across several tasks. Compared with vanilla OPD, D-OPD improves OCR from 0.90 to 0.94, PickScore from 22.80 to 23.31, and GenEval from 0.89 to 0.96. It also improves all reported metrics on MultiEvalBench. This provides useful evidence that the gains extend beyond a single task or training reward.

  2. The method is simple, and the ablations support its practical design. Using one negative teacher gives nearly the same results as using all teachers, while restricting the contrastive objective to early steps further improves convergence in training iterations. The five-teacher experiment also shows that the approach can accommodate additional experts.

Critical weaknesses
  1. The experiments do not fully separate teacher discrimination from changes in loss weighting. With the stop-gradient operation, the positive-teacher KL gradient has a coefficient of 1/τ1/\tau. At the chosen τ=10−3\tau=10^{-3}, this also strongly changes the weighting of early steps relative to later steps and the reference-model regularizer, unless another normalization is applied. The temperature sweep is useful, but it does not isolate these effects. This leaves some uncertainty about how much of the gain comes specifically from pushing away the other teachers.

  2. The efficiency evidence is mainly based on training iterations. D-OPD evaluates an additional teacher during early steps, so fewer iterations do not directly establish lower training cost than vanilla OPD. The dual-teacher and early-window strategies clearly reduce work compared with the full D-OPD objective, but wall-clock measurements are needed to establish the practical speedup over the baseline.

Questions for authors
  1. Is the cross-entropy loss rescaled or normalized before being combined with the later-step OPD loss and reference KL regularizer? Do you have an existing comparison that keeps these weights matched while removing the negative-teacher term?

  2. From the existing training runs, what are the time per iteration and total time to reach a common performance level for D-OPD and vanilla OPD, using the same hardware?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.