acceptodds
Withdrawn from ICLR 2027

Rethinking On-Policy Distillation in Diffusion Models: Phenomenology and Empirical Insights

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) consolidates diffusion-model specialists into a shared student by supervising its own denoising trajectories. Yet its mechanisms remain underexplored: What does OPD transfer? What determines transfer at student-visited states? When is distillation worthwhile relative to direct joint optimization? We study three representative OPD methods on two backbones and thirteen routes through budget-controlled comparisons, held-out behavioral evaluation, frozen-trajectory interventions, temporal supervision and teacher-source experiments, and comparisons with direct joint reinforcement learning. First, OPD consolidates specialist gains selectively: DanceOPD leads three of four equal-time settings, whereas DiffusionOPD leads three of four equal-data settings, with winners changing in 17 of 52 matched route comparisons. In the Z-Image two-interface equal-data setting, held-out image-editing and multi-image scores improve by approximately 3.4% and 5.8% over the exact initialization. Second, transfer depends on timing and teacher source: query-matched disagreement weighting improves held-out image editing and multi-image generation by approximately 10.3% and 2.8% over uniform weighting, yet fixed low-noise supervision leads on ten of thirteen routes. Moreover, a reinforcement-learning teacher that scores 6.1% above its supervised-fine-tuning counterpart produces a student that scores 2.9% lower in the design case. Third, under UnifiedReward, direct joint reinforcement learning improves general text-to-image, general image-editing, and held-out image-editing scores by approximately 5.5%, 6.6%, and 93.3% over initialization. Equal-data DiffusionOPD is 2.6% better than direct joint reinforcement learning on general image editing but 16.9% and 24.1% worse on general text-to-image and held-out image editing, respectively. We hope these findings inform future OPD design and use by the community. All resources will be publicly released, including code, checkpoints and datasets.

Primary area:
generative models
Keywords:
on-policy distillation, diffusion model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_B565OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies OPD in diffusion models, asking what OPD transfers, what determines transfer at student-visited states, and when distillation is worthwhile relative to direct joint optimization. It evaluates three representative OPD methods on two backbones and thirteen routes through budget-controlled comparisons, held-out behavioral evaluation, frozen-trajectory interventions, temporal-supervision and teacher-source experiments, and comparisons with direct joint reinforcement learning. The results show that OPD consolidates specialist gains selectively, method rankings depend on the training budget, and a higher-scoring teacher can produce a weaker student.

Critical strengths

The paper conducts an extensive empirical evaluation of existing OPD methods across two backbones, thirteen task routes, and multiple training configurations, including equal-time and equal-data comparisons. This provides useful practical evidence for understanding their effectiveness and limitations beyond a single task or configuration.

Critical weaknesses

The paper does not sufficiently synthesize its extensive experimental observations into a clear set of contributions. The narrative emphasizes individual results and numerical comparisons, while the insights and their implications for understanding OPD remain underdeveloped. This makes the conceptual contribution difficult to assess.

Questions for authors

Could the authors clarify the paper’s central insights and explain how the experimental evidence supports each of them and establishes the main contributions?

Reviewer_PRd6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies on-policy distillation (OPD) for consolidating diffusion-model specialists into a shared student. It evaluates three representative methods, DanceOPD, FlowOPD, and DiffusionOPD, on two backbones and 13 routes covering text-to-image generation (T2I), image editing (I2I), and multi-image composition (MI2I). The study is organized around three questions.

Q1: What does OPD transfer? The authors compare specialist retention under equal-time and equal-data budgets and evaluate held-out behavior relative to teacher resemblance. They find that OPD transfers specialist gains selectively, method rankings depend on the training budget, and teacher resemblance does not necessarily predict task performance.

Q2: What determines transfer at student-visited states? Through frozen-trajectory interventions, temporal supervision, and SFT-versus-RL teacher comparisons, the paper shows that transfer is sensitive to correction direction, timing, and teacher source. Fixed low-noise supervision performs best on most evaluated routes, while a stronger teacher does not necessarily produce a stronger distilled student.

Q3: When is distillation worthwhile relative to direct joint optimization? The authors study reward transfer within T2I and cross-interface consolidation. OPD can transfer individual reward preferences, while direct joint RL preserves broader cross-task behavior better in the tested settings

Critical strengths

S1. Broad and well-motivated empirical study. OPD for diffusion/flow-matching models is a growing research direction, and the paper provides a useful empirical map of the problem. The three research questions are clear and well motivated, and the evaluation spans multiple OPD methods, backbones, task families, budget definitions, teacher sources, and held-out evaluations.

S2. Several observations challenge common assumptions about OPD. In particular, teacher resemblance does not necessarily imply better task performance, the best method can change between equal-time and equal-data comparisons, maximum-disagreement timing does not outperform fixed early intervention, stronger teachers do not necessarily transfer better, and direct joint RL is an important baseline when broad capability preservation is required. These are useful observations for researchers designing and evaluating OPD systems

Critical weaknesses

W1. The analysis identifies many factors affecting OPD, but provides limited insight into what actually enables successful transfer.

The paper shows that OPD depends on budget, timing, teacher source, backbone, and task, but many conclusions remain “it depends.” For example, method rankings flip between equal-time and equal-data settings without isolating which differences in query density, rollout, objective, or regularization cause the reversal. Similarly, SFT vs. RL teacher rankings reverse across tasks without identifying what predicts teacher transferability.

More fundamentally, identifying several failure modes does not explain what makes OPD succeed. The paper shows that teacher quality, resemblance, and maximum disagreement are not reliable predictors, but does not provide an alternative predictive principle or an actionable strategy that consistently improves OPD. Low-noise supervision is a useful heuristic, but overall I view the work as a valuable first-step phenomenological characterization rather than a sufficient answer to the paper’s central question of what determines transfer.

Questions for authors

Q1. Budget dependence: Which concrete component of DanceOPD, FlowOPD, or DiffusionOPD explains the ranking changes between equal-time and equal-data budgets? Can query density, rollout distribution, objective, or regularization be isolated experimentally?

Q2. Teacher transferability: Is there any measurable property of the teacher/student pair that predicts whether an SFT or RL teacher will transfer better? This seems especially important because the preferred teacher source reverses across the studied tasks.

Q3. Backbone dependence: Can the authors analyze why some conclusions or rankings differ between SD3.5-M and Z-Image and whether these differences are statistically meaningful?

Q4. Actionable OPD improvement: Most importantly, can any of these findings be turned into a teacher-selection, timestep-selection, weighting, or training strategy that consistently improves an existing OPD baseline? Such a result would substantially strengthen the claim that the study explains what determines successful transfer rather than primarily cataloguing sensitivities and failure modes.

A deeper analysis along one of these directions would substantially increase the value of the current empirical study, especially Q4.

Reviewer_89dcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how multi-teacher on-policy distillation consolidates generation and editing capabilities in diffusion models. It compares three OPD methods across two backbones and 13 tasks, exploring training budgets, supervision timesteps, and teacher selection. Additional experiments investigate teacher corrections along student trajectories and compare distillation with direct joint RL and parameter merging.

Critical strengths

S1 | The paper provides several useful empirical observations in multi-teacher OPD, e.g., low-noise timesteps are beneficial, training budgets affect OPD method rankings, and teacher performance alone is insufficient to assess transfer quality.

S2 | The evaluation covers three recent OPD methods, two backbones, and 13 generation and editing tasks, providing a reasonably broad basis for the conclusions.

Critical weaknesses

W1 | The paper is difficult to follow because it reads as a collection of experiments with different setups and extensive result discussions. Figures and tables often require explanations scattered across the paper and appendix. The paper would benefit from significant restructuring around a few central findings, with clearer descriptions of each experiment’s purpose and implications.

W2 | Several findings are useful observations, but their implications for improving OPD remain limited. For example:

Section 4.3 demonstrates mixed conclusions with limited explanation and understanding how to select a suitable teacher in a particular case.

Section 4.1 studies corrections to frozen trajectories, but does not show how this affects OPD training. Therefore, it remains unclear what this experiment gives in the paper context.

The clearest training recommendation is low-noise supervision and it was already supported by DanceOPD (Section 4.3).

The paper would benefit from prioritizing its strongest new findings, clarifying their practical implications, and moving secondary experiments to the appendix.

Questions for authors

Could the authors summarize the key experimental findings and explain how they can guide practical improvements to OPD?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.