acceptodds
Under review as a conference paper at ICLR 2027

GOJO: Geometry-preserving Joint Optimization for On-policy Distillation in Diffusion Models

Abstract

In post-training of diffusion models, on-policy distillation (OPD) offers an effective way to combine skills from multiple teachers into a single student. While these teachers are obtained through multiple routes, such as supervised fine-tuning (SFT) or reinforcement learning (RL), a unified regression objective is used to match the velocity. Despite its simplicity, this regression objective can lead to unexpected velocity conflicts among teachers' supervision signals and drift in the student model prior, which causes catastrophic forgetting and teacher matching failure. In this paper, we address this limitation through a local probability-reweighting formulation, where we assign credits to each sampled candidate velocity at student-induced states and update the velocities based on the credits. The credits are constructed through three factors: 1) target teacher agreement that directly guides the student velocity; 2) probability redistribution cost that measures how much the student's geometry is perturbed; and 3) mutual teacher damage that quantifies the impact between teachers. We then integrate these three objectives into a single optimization framework, and introduce GOJO, Geometry-preserving Joint Optimization for multi-teacher on-policy distillation. The resulting optimization problem yields a closed-form solution for the optimal target velocity, which can be used for velocity regression in training. Experiments on SD3.5-Medium and Qwen-Image show that GOJO achieves superior performance in Multi-teacher Distillation with an average 10.5% improvement across tasks on hard-routing OPD and 5.9% on soft-routing OPD, while significantly mitigating the student model prior and avoiding teacher conflicts.

Primary area:
generative models
Keywords:
Diffusion Models, On-policy Distillation, Post-training, Text-to-Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_J9C5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GOJO, a framework for multi-teacher on-policy distillation of diffusion models. Instead of directly regressing to teacher velocities, GOJO constructs a target that balances teacher agreement, a covariance-based penalty on student displacement, and increases in other teachers' matching errors. The target is obtained by solving a small dual problem followed by an analytical velocity update. Experiments on SD3.5-Medium report improved capability transfer, while Qwen-Image experiments motivate the problem of teacher interference.

Critical strengths
  • The paper studies a meaningful problem. Combining specialist teachers can create conflicting supervision, and accounting for both transfer and interference is a useful direction.
  • The target construction is technically interesting and keeps student training as a simple regression problem. The appendix provides substantial details on covariance estimation, singular covariance, and the numerical solver.
  • The reported SD3.5-Medium results are good, and the covariance and teacher-protection ablations help characterize the method.
Critical weaknesses
  • My biggest concern is attribution in the comparisons with complete OPD recipes. Appendix F, Table 3 shows differences in query construction and teacher guidance: GOJO uses eight endpoint re-noising queries, whereas the baselines use rollout states and different time-sampling or CFG conventions. Appendix E.5 further introduces RMS compensation and a separate base-unconditional consistency loss. These choices can affect both reward and drift. Although Table 2 provides target ablations, the headline comparisons and behavior diagnostics need a baseline with the same queries, guidance, outer losses, and optimization budget, changing only the proposed target.
  • The practical covariance is not the query-conditional geometry used in the theory. Appendix E.3 pools residuals across prompts, spatial locations, and time buckets, and explicitly shows that the estimate contains conditional-mean error. Appendix E.6 also notes that the old student's velocity need not equal the required conditional mean. This makes the implemented penalty a surrogate. The isotropic ablation is useful, but its update RMS differs substantially from the diagonal and channel versions. A scalar-shrinkage baseline with matched correction magnitude and outer weighting is needed to determine whether the improvement depends on meaningful anisotropic geometry.
  • The probability-reweighting derivation needs correction and a clearer scope. A valid reweighted distribution requires 1+A >= 0, but the minimum-energy credit in Appendix C.1.2 is optimized without this constraint. Appendix D.1.1 acknowledges that the resulting linear credit may violate it. Thus, the claimed equivalence is to a relaxed quadratic credit problem, rather than generally to valid probability redistribution. In addition, Eq. 9 and Proposition 4 miss the factor 1/2 in the local expansion KL = (1/2) E[A^2] + higher-order terms. This constant can be absorbed into lambda, but the derivation should still be corrected and distinguished from the stronger probability interpretation.
  • The evidence for mutual-teacher protection is internally inconsistent. In Table 2, enabling protection changes diagonal H+ from 0.007 to 0.053 and channel H+ from 0.013 to 0.081, although lower is better. Figure 4 and Sec. 5.3 describe approximately the opposite direction and claim reductions of 86.0% and 84.3%. Different checkpoints or protocols might explain this, but they are not identified clearly. This discrepancy directly affects the empirical support for one of the central components.
  • The validation is narrower than the framing suggests. Qwen-Image is used for the motivating conflict analysis, while the main GOJO transfer experiment uses three RL-trained SD3.5-Medium teachers. This leaves the motivating SFT/RL mixture insufficiently tested. Moreover, Table 2 averages repeated monitoring checkpoints from one seed, which does not establish training-run uncertainty. Absolute training time and teacher-query counts would also help interpret the reported 1.09x relative cost, especially when hard routing still queries other teachers for protection.
Questions for authors

Please see above weaknesses.

Reviewer_Q97qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes geometry-preserving joint optimization for multi-teacher on-policy distillation (GOJO), which is a diffusion distillation method for combining specialized teacher diffusion models into a single student model. GOJO contructs an OPD-style regression target that jointly optimizes (1) teacher agreement, (2) probability redistribution cost, and (3) mutual teacher damage. This accounts for the student's local probability geometry and conflict among teacher models when consolidating teacher capabilities into the student model. Experiments report GOJO's task performance in the multi-teacher distillation setting, compared with DiffusionOPD and DanceOPD baselines.

Critical strengths
  1. Combining specialized teacher diffusion models into a single student model while preserving student geometry and limiting interference between teachers is an important research problem. The motivation is reasonable and the proposed method provides a practical solution. The presentation is clear and easy to follow, which provides sufficient background knowledge and guides readers to the problem through a motivating conflict analysis.
  2. The final optimization objective is generally interpretable and tractable, which leads to improved empirical task performance over DiffusionOPD and DanceOPD on SD3.5-Medium.
Critical weaknesses

The most critical weakness is that the paper defines soft-routing OPD as averaging all teachers at the same query in Appendix E.2 and claims that the proposed GOJO framework accommondates soft-routing OPD like DiffusionOPD. However, this is not how the original version of DiffusionOPD is implemented: as I understand, the original DiffusionOPD supervises each task’s prompts with its corresponding teacher and accumulates task losses before updating, which is different from the definition in the paper. Therefore, the claimed reduction to DiffusionOPD does not seem to hold. Furthermore, this raises a serious question about whether the DiffusionOPD baseline in the experiment is implemented correctly.

I also have other concerns which are listed in the Question section below.

Questions for authors
  1. How is the DiffusionOPD baseline implemented in the experiment? Does it supervise each task’s prompts with its corresponding teacher and accumulates task losses before updating, or average all teachers at the same query? Under what assumptions does GOJO recover DiffusionOPD?
  2. Could the authors implement a tuned scalar-shrinkage baseline with comparable correction norms under the same training recipe and compare its performance to GOJO?
  3. How much of the drift reduction in Fig 6 remains when guidance, queries, RMS weighting and unconditional consistency are held identical across all methods? What happens when the explicit unconditional consistency loss is removed?
Reviewer_uEutOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces GOJO, a geometry-preserving joint optimization framework for multi-teacher on-policy distillation (OPD) in diffusion models. The authors identify two limitations of conventional velocity regression: conflicting supervision from multiple teachers and student model prior drift. To address these issues, GOJO formulates distillation as a joint optimization problem incorporating teacher agreement, probability redistribution cost, and mutual teacher damage. The framework derives a closed-form target velocity using covariance-based approximations of the student's local probability geometry. Experiments on SD3.5-Medium demonstrate improvements over DiffusionOPD and DanceOPD across multiple image generation tasks, including GenEval, OCR, and PickScore.

Critical strengths
  1. The paper addresses a clear and relevant problem in multi-teacher distillation, which is both theoretically and practically motivated.
  2. The proposed framework provides a principled optimization formulation and derives a closed-form solution for the target velocity.
  3. The method introduces relatively modest computational overhead compared to existing OPD approaches, making it practical for large-scale diffusion models.
  4. It provides insightful analyses of teacher conflicts and model prior drift, offering empirical evidence to support the proposed approach.
Critical weaknesses

Insufficient validation of mutual-teacher conflict minimization. The ablation results in Table 2 suggest that most performance gains originate from geometry preservation, while the additional improvements from conflict minimization are relatively marginal. I encourage the authors to evaluate GOJO under varying degrees of teacher conflict and report individual teacher capability retention. In particular, comparing geometry-only and full GOJO under increasingly conflicting teacher combinations would better demonstrate the necessity of the proposed mutual-teacher protection mechanism.

Questions for authors
  1. Vary the degree of teacher conflict. Construct low-, medium-, and high-conflict teacher combinations, measured using the velocity or gradient conflict metrics already introduced in Section 3. Compare geometry-only GOJO with full GOJO across these settings.

  2. Evaluate individual teacher capability retention. Measure each teacher's task performance before and after incorporating other teachers into the student. This would directly test whether mutual-teacher conflict minimization prevents negative transfer.

Reviewer_782MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-teacher on-policy distillation (OPD) for flow-matching text-to-image models, arguing that plain velocity regression can cause teacher conflicts and student prior drift. GOJO instead regresses the student to a closed-form target that balances teacher agreement, probability redistribution cost, and mutual teacher damage. Experiments distill three RL teachers (GenEval, OCR, PickScore) into SD3.5-Medium and compare with DiffusionOPD and DanceOPD.

Critical strengths
  1. The paper offers a principled formulation whose target has a closed form, reduces to standard OPD when both penalties are disabled, and adds little overhead (1.09× DanceOPD’s wall-clock time).
  2. The ablations suggest that the redistribution term is effective within the proposed recipe, notably improving GenEval and OCR under hard routing (Table 2: 0.801→0.873 and 0.786→0.948). The final model also achieves the best in-domain scores among the compared OPD methods in Table 1.
Critical weaknesses
  1. The motivating analysis focuses on interference between SFT- and RL-derived teachers (Prop. 1, Sec. 3), but the main experiments use three RL teachers. For RL teachers, the paper’s own analysis (Eq. 7) suggests that the shared component is already removed by advantage centering. In addition, GOJO results on the Qwen-Image SFT+RL setting appear to be limited to the qualitative examples in Fig. 1, so it remains unclear whether GOJO mitigates the failure mode shown in Fig. 2.
  2. The covariance is pooled over prompts, spatial positions, and time intervals (App. E.3), so it is unclear how much of the query-specific local geometry that motivates the method is retained in practice. A comparison with a simple damped target of matched step size under the same recipe, e.g., v_old + β(v_T − v_old), would help clarify how much of the gain comes from the geometry term itself.
  3. Some supporting analyses are difficult to interpret. The drift comparison in Fig. 6 is measured on the unconditional branch, which GOJO’s recipe also regularizes through a separate consistency loss (App. E.5). Separately, Table 2 and Fig. 4 appear to report the same H+ values with opposite labels.
  4. The reproduced DiffusionOPD stays close to or below the base model with CFG on GenEval and OCR (0.66 vs. 0.64; 0.47 vs. 0.56), while DanceOPD reaches 0.82/0.92 with the same teachers. This raises the question of whether the baseline is well tuned. The paper also describes DiffusionOPD as averaging all teachers at each query (Sec. 4.2, App. E.2), whereas in Li et al. (2026) each task’s prompts are supervised only by that task’s teacher.
Questions for authors
  1. How was DiffusionOPD implemented: with per-task routing, or by averaging over all teachers? Did the baselines also use the consistency loss and RMS compensation?
  2. Which of Table 2 and Fig. 4 has the correct H+ labels?
  3. Would it be possible to include the matched-step damping baseline, the drift of GOJO’s recipe with λ = γ = 0?
  4. Could you report quantitative GOJO results in the Qwen-Image SFT+RL setting, since this is where the motivating failure is demonstrated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.