- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper targets multi-task post-training of text-to-image flow models by using OPD from several task-specific teachers such as OCR, PickScore, and GenEval. The authors argue that original OPD leaves the student's rollout indistinguishable across teachers. The proposed InfoNCE-style cross-entropy loss that raises the probability of the task teacher and implicitly pushes the student away from the other teachers. Evaluation results of proposed D-OPD demonstrate faster reward convergence than original OPD and multi-reward GRPO.
Critical strengths
-
The method is simple yet efficient and easy to adopt on existing on-policy distillation for many application. The proposed DT and EW address the obvious cost of evaluating all teachers at every step.
-
The empirical improvement is significant. Convergence is much faster than vanilla OPD and MR-GRPO, consistently across 2, 3, and 5 teacher settings.
Critical weaknesses
-
The evaluation results are statistically unsupported. the high-variance and unstable nature of reinforcement learning, multi-seed run and significance test are required for evaluation. However, there are no multi-seed statistic, error bars, or confidence interval. Also test-split metrics are the same reward models used to train the teachers, so they cannot detect reward hacking. Multi-seed run and statistical report and qualitative results and human evaluation might address this concern.
-
DT and EW are motivated by computational cost, but there are no wall-clock time, NFE count, or memory figures are reported. Also convergence speed is measured per training iteration. This may not mean real efficiency of training cost.
Questions for authors
- Please report results over multiple seeds with error bars, plus wall-clock time and NFEs per iteration for vanilla OPD, D-OPD w/o DT/EW, D-OPD w/o EW, and full D-OPD.