acceptodds
Under review as a conference paper at ICLR 2027

MSE Training Induces Diffusions That Are Nearly Flows

Abstract

In the DDPM paper, Ho, Jain, and Abbeel introduced two reversible diffusion processes parameterized by a noise schedule—a generator and an oracle process that the generator learns from—and derived a formula for the Kullback-Leibler divergence (KL) between these processes in the form of a time-weighted Mean Squared Error (MSE). However, they empirically found that omitting the weights improved performance on image synthesis benchmarks—a result later corroborated by many studies. More recently, removing the stochastic component during generation time has proven effective. (1) In this work, we provide a theoretical justification for these practices. We consider a broader class of diffusion processes (not necessarily reversible) parameterized by a noise schedule and a diffusion size b that share the same marginals. Since the weight associated with the MSE depends on b, omitting the weight is equivalent to solving the equation weight(b)=1, which yields a unique “MSE-diffusion”. For SOTA models, we checked that b is close to zero—the learned MSE-diffusion is nearly a flow—and we confirm this observation by comparing generators on ImageNet 512×512. These results suggest that flows beat reversible diffusions because training of SOTA models is an implementation of KL minimization for MSE-diffusions, which are nearly flows. (2) Moreover, we obtain a novel representation of the diffusion state as the sum of an explicit linear component, an unweighted pathwise integral of the denoiser, and a noise term. This representation offers the advantages of DPM-solvers while enabling the use of classical ODE methods.

Primary area:
generative models
Keywords:
generative models, diffusion models, matching flows, computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GmFgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the relationship between the MSE training objective and the stochasticity of sampling process, and try to answer why the deterministic sampling process can be supported by MSE training objective.

Critical strengths
  1. this paper studies a fundamental problem between diffusion model and flow matching, i.e. the relationship between stochastic training objectives and deterministic sampling process, which I think has been solved in some researches, like DDIM [1].

[1] Denoising Diffusion Implicit Models, 2010

Critical weaknesses
  1. the Λ\Lambda in the MSE optimization equation below equation 1 is not defined.
  2. I think the proposed problem that the DDPM objective can support DDIM sampling has already been theoretically solved and empirically verified in the following researches [1-4].

[1] Denoising Diffusion Implicit Models, 2010

[2] Score-Based Generative Modeling through Stochastic Differential Equations, ICLR 2021 oral

[3] Elucidating the Design Space of Diffusion-Based Generative Models, NIPS 2022

[4] The Principles of Diffusion Models, arxiv 2025

Questions for authors

see weaknesses.

Reviewer_pxu2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper reinterprets the standard MSE training objective as a KL minimization for a selected family of diffusion processes. It argues that the resulting processes are nearly deterministic for several modern approaches, offering an explanation for the effectiveness of the flow-based sampling. It also introduces a reformulated sampler/solver, providing some empirical validation of the proposed framework.

Critical strengths
  • This paper addresses a relevant question: how practical MSE training objectives relate to the stochasticity during sampling
  • The authors present an explicit algebraic connection between loss weighting and diffusion rates, offering a potentially useful perspective on existing models
  • The diffusion state equation reformulation treats the linear component analytically, facilitating the use of standard integration methods, and appears to be new in the context of generative models.
Critical weaknesses
  • The paper presentation obscures and over-complicates the central idea through unconventional notion and lengthy derivations. The manuscript would drastically benefit from a full rewrite focused in improving the clarity of exposition, simplifying the notation, providing clear and simpler examples, and improving the wording. I found the paper very hard to follow.
  • Its a bit unclear about the novelty of the paper relative to existing diffusion/flow-matching pipelines. The authors could try to summarize different approaches more clearly on the appendix. The current way that this is structured on the paper is muddy and confusing. The second paragraph of the experiments attempts to do that, but again, hard to follow.
  • The claim that the proposed MSE training induces a nearly unique and nearly deterministic diffusion model depend on additional choices. For example, the solution of the proposed MSE equation picks the smaller root through an additional penalty interpretation. This is also what induces the nearly flow regime, which is not super clear on the paper presentation.
  • The experimental evidence is limited relative to the claim of explaining why flows outperform diffusions. Matching training objectives does not directly establish better sample quality and diversity. The authors would have to support their claims with more detailed experiments and ablations. There's only one plot and a table in the whole paper.
Questions for authors

Why should the smaller root be preferred, given that both solutions yield the same diffusion loss? In what sense is the proposed process uniquely induced by MSE training?

Reviewer_faseOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper essentially proposes a recipe to derive an inference time sampling schedule based on the training conifiguration of the generative model. The paper shows numerical examples in an implementation of the recipe, and performs experiments on sampled quality (FID) when they pair the recipe with existing SOTA models.

Critical strengths
  1. The paper provides an interesting and reverse angle to decide sampler based on the training configuration of the underyling generative model, while previous works establish the theoretical framework the other way around.
  2. The paper provides related, though not direct, explanation of why flow is better than diffusion.
Critical weaknesses
  1. The recipe only depends on the training configuration but not the model itself. This might be fine for a completely mature model, but it would hard to tell if a suboptimal generation is because of the underlying generative model itself even if the recipe is used.
  2. The experiment scale is really not large enough to be convincing. The recipe can in theory be used on non-SOTA model too, and that might be where more evidence can be shown.
  3. A good test would be to train a model by ELBO-weighted MSE faithfully, where the theory predicts that reversible diffusion would be competitive to a flow based sampling.
Questions for authors
  1. The table includes SiD2 which is nearly reversible. Can you provide explanaiton based on the theory? Other please see weakenss.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.