acceptodds
Under review as a conference paper at ICLR 2027

Guidance Without Labels: Flow Matching from Coupled Gaussian Mixtures

Abstract

Flow matching models are typically trained to transport a Gaussian source to the data distribution, with source and target samples paired independently. We replace the source with a fitted Gaussian mixture and couple each data point to a mixture component through an entropic optimal transport problem. We show that the resulting transport reduces to the standard Gaussian-source problem up to a guidance-like correction, which enables the reuse of a pretrained velocity field. This correction is a closed-form function of the fitted mixture plus a term that we learn with a small correction network. We test our model on published checkpoints for CIFAR-10 and CelebA-HQ. It improves FID at every step count, and on CelebA-HQ it reaches the quality of the baseline with about half the steps. The closed-form term alone, with no training, already delivers most of this improvement; the small correction network accounts for the rest.

Primary area:
generative models
Keywords:
flow matching, optimal transport, Gaussian mixture models, classifier guidance

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_bUb8OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The proposed paper fit an isotropic gaussian mixture to the data and couples data points to mixture components through posteriors. A time-transformation and guidance correction relates the resulting flow to a gaussian source, allowing reuse of a frozen pretrained backbone. The practical correction combines an analytical GMM term with a small learned residual. Experiments on a checkerboard example, CIFAR-10, and CelebA-HQ demonstrate improved FID across the reported Euler budgets.

Critical strengths
  • Proposition 5 provides an explicit GMM correction that makes a structured source usable with an existing Gaussian-source checkpoint. That is a strong technical contribution, as it enables a practical correction while keeping the backbone frozen.
  • The trained variant improves every reported Euler FID point on both datasets. The checkerboard ablation help explain the mechanism, and the analytical variant provides substantial gains at low step counts after fitting the mixture.
Critical weaknesses
  • The specific advance over existing guidance matching is not fully clear. On the Guidance of Flow Matching (ICML 2025), is a closely related omitted reference. Their Appendix A.4 gives the same ((1-t)/t) guidance coefficient. Under the paper’s component-marginal condition, their MRGM loss, Eq. (45), yields Eq. (21) after averaging over components. The new contribution therefore lies in the GMM-specific decomposition and parameterization, which is not new.

  • Appendix B.3, invokes Proposition 3 for the Wϕ=0W_\phi=0 results, although that proposition assumes the exact conditional field. Equations (19)–(20) replace qtq_t by qtrefq_t^{\mathrm{ref}} and omit the remaining correction. Even with an exact base velocity,

Questions for authors
  • Contribution of the analytical term: With the same frozen backbone, GMM responsibilities, correction-head architecture, how does Eq. (19) compare with learning the entire additive correction without its analytical term?

  • Also, Please report FID and sampling time, and also clarify the relationship to MRGM's objective.

  • Remaining approximation error: On the checkerboard example, what endpoint distribution error remains for the untrained and trained variants when the solver is run to numerical convergence? Can authors distinguish field approximation from discretization error?

  • Reliability and scope of the speedup: Please report uncertainty and end-to-end timing for the CelebA-HQ comparisons of 64 versus 128 steps.

  • Can authors please Include the GMM-fitting cost alongside the correction-network parameter count.

  • The 1,000-sample nearest-neighbor analysis in Section 5.4 supports no detected increase in near-copy behavior. It does not establish a general absence of memorization. I am interested to see how authors defend their claim/s here.

  • The statement on p. 2 that classifier guidance necessarily at least doubles per-step cost is too strong. Its overhead depends on the classifier architecture and input-gradient computation.

3 comments

We thank you for the careful review. The changes to the paper are in red in the revised PDF, whose equation and table numbers we use here (a new Eq. (11) and a new Table 1 shift the later numbers by one).

Relation to prior guidance work. We now cite Feng et al. (2025). In their framework, each slice p1(k)p_1^{(k)} is p1p_1 reweighted by e−J(x1)=Γ(k∣x1)e^{-J(x_1)} = \Gamma(k \mid x_1), Eq. (12) is their guidance field and Eq. (22) is their MRGM loss averaged over components. For our coupling the guidance is exact, a case they treat only approximately (their Section 3.1), and closed form up to one scalar function (Eq. 19), with an energy that comes from the fitted mixture and needs no labels. We use it to improve unconditional generation, not to steer the target.

Appendix B.3 and Proposition 3. You are right, Proposition 3 assumes the exact conditional field, and the untrained model replaces qtq_t by qtrefq^{\mathrm{ref}}_t and drops the second term of Eq. (19). We updated the paper accordingly.

Contribution of the analytical term, FID and sampling time. We trained the same head without the analytical term, with mkm_k and qtrefq^{\mathrm{ref}}_t still as inputs. Its sampling time is that of the trained model, 3 percent above the baseline on CIFAR-10 and 5 percent on CelebA-HQ. Its FID is

FID at Euler steps1248163264128256
CIFAR-10, trained (Table 3)151.3387.0130.5214.108.085.193.913.212.95
CIFAR-10, no analytical term154.7587.3229.7913.437.615.113.813.102.85
CelebA-HQ, trained (Table 3)182.97115.0487.3858.9033.4016.6911.249.809.42
CelebA-HQ, no analytical term220.12130.6795.8064.7638.7421.4913.9811.4211.13

On CIFAR-10 this variant is slightly better from 4 steps on, but on CelebA-HQ it is worse at every step count, and at 256 steps even worse than the Gaussian baseline (10.01). We think that, due to the higher dimension, the analytical term finds the correct posterior more quickly on CelebA-HQ. We report this variant in Appendix B.3 of the revision.

Approximation error on the checkerboard. Sliced 2-Wasserstein distance between 250k endpoints of the four models of Figure 1 and the checkerboard density, with Euler at the step counts of Figure 1 and RK4 at 1024 and 2048 steps.

Sliced W2W_2Euler 4Euler 16Euler 64Euler 256RK4 1024RK4 2048
Gaussian0.8030.2270.0880.0590.0510.051
GMM uncoupled0.8050.2300.0900.0600.0530.053
GMM coupled untrained0.1230.0500.0390.0380.0380.038
GMM coupled trained0.1110.0350.0250.0240.0240.024

The two RK4 columns agree to four decimals, so they measure the exact flow of each field, and the rest of each row is discretization error. The coupled models learn a better density, their exact flow ends closer to the target, and they reach their own limit in far fewer steps. Two independent exact samples are at distance 0.017, a floor that no model can beat. We added this to Section 5.1 of the revision.

Reliability and timing of the speedup. FID over three sets of 10k samples (mean and standard deviation) and time per sampling step on a V100 with a batch of 16 images.

CelebA-HQEuler stepsFIDtime per step
Gaussian baseline12811.41±0.1711.41 \pm 0.17536 ms
GMM trained6411.34±0.1011.34 \pm 0.10563 ms
Gaussian baseline2569.99±0.029.99 \pm 0.02536 ms
GMM trained1289.71±0.089.71 \pm 0.08563 ms

The trained coupled model reaches the quality of the baseline with half the steps, in 52.5 percent of the time.

Cost of the mixture fit. On one V100, fitting the mixture takes 16 minutes on CIFAR-10 (K=4096K = 4096) and 76 minutes on CelebA-HQ (K=1024K = 1024), and training the correction network 1.4 and 17 hours. Timing one training iteration of each base network on the same GPU and multiplying by its released iteration count gives 207 and 4540 hours for the base networks, so our additions cost under 1 percent of their training. We added this comparison to Section 5.2 of the revision.

Memorization. We agree that claiming no memorization is too strong. We now only claim that our model does not memorize more than the Gaussian baseline, in Section 5.4 and in the conclusion. We also repeated the experiment with 10000 samples per model on both datasets for more statistical significance, and updated Table 4 and the figures accordingly. With 10000 samples, the coupled model remains slightly farther from the training set than the baseline on CIFAR-10 and as far on CelebA-HQ. Moreover, the baseline produces five near copies of a car that appears several times in the CIFAR-10 training set, and the coupled model produces none.

Cost of classifier guidance. You are right, we updated the paper accordingly.

Dear Authors,

Thank you for the response and the additional experiments. Some of my earlier concerns (in Questions and Weakness section) have been addressed, and I appreciate the authors' efforts. However, I am still not satisfied with the authors' response to W1 about the novel contributions beyond the related work. Therefore, I would like to keep my original rating for now.

We thank you for your reply. Could you tell us what remains unsatisfactory in our answer to W1, so that we can address it before the end of the discussion?

To restate it briefly, the only element we share with Feng et al. is Eq. (12) of the revised PDF. It is classifier guidance, as in Song et al. (2021) and Dhariwal and Nichol (2021), written for flow matching in Eq. (4.90) of Lipman et al. (2024), which predates Feng et al. The paper derives it in two lines in Appendix A.2, the revision cites these works next to it, and we do not claim it.

Everything else differs. We change the source and the coupling, which is the mixture posterior and the optimal entropic coupling (Propositions 1, 2 and 4), and the target stays p1p_1 (Proposition 3), whereas Feng et al. keep the Gaussian source and reweight the target. Our guidance needs no Monte Carlo estimate and no learned normalizer, since it is closed form up to one scalar function (Proposition 5), and it beats the baseline without any training (Table 3). Its energy comes from the fitted mixture and needs no labels.

Reviewer_WiNFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Having built upon recent works of GMM as the source distribution with mode-dependent coupling, the paper proposes a sampling-correction method to pretrained flow-matching models trained with isotropic Gaussian source distribution and independent coupling, by fitting a GMM on the target distribution and using it to start and guide the transport. The theoretical analysis shows the fitted GMM's posterior probability (i.e., the soft assignment of each data point to its Gaussian component) provides an exact entropic optimal coupling and under which the velocity is the pretrained model's velocity plus the gradient of the component classifier, which is explicit from the fitted GMM. The empirical results show the proposed method improves inference efficiency by reducing the NFEs to obtain similar generation fidelity on CIFAR-10 and CelebA-HQ datasets over some released pretrained models.

Critical strengths
  • From the lens of EOT that encourages good properties of a convex problem, the optimal entropic coupling becomes GMM's posterior responsibilities, the soft assignments from the E-step of EM. Though the paper itself says this is a known reformulation (Rigollet & Weed 2018; Vayer & Lasalle 2025), its application to justify the data coupling for flow matching is very refreshing.
  • With the η=2τ2\eta=2\tau^2 trick and the shared-scale isotropic covariance across all components, the coupling derived from the EOT's KL penalty becomes a softmax, which is exactly the GMM posterior. That enables re-using a pretrained flow-matching model from isotropic Gaussian source (which is typically the case), and re-using of the GMM posterior probability for data coupling, which also makes the classifier (i.e., mode) guidance nearly closed-form.

References

  • Rigollet & Weed (2018) Entropic optimal transport is maximum-likelihood deconvolution
  • Vayer & Lasalle (2025) A note on the relations between mixture models, maximum-likelihood and entropic optimal transport
Critical weaknesses
  • The writing flows naturally once the readers buy modifying the source distribution to GMM is a good idea, but it does not establish enough intuition that why do we want GMM source at the first place. Prior works the paper cites use GMM source because the target distribution is multimodal (i.e., multi-mode), but I am not convinced that both image datasets used in this paper are multimodal.
  • Conceptually it's convenient to "move" the solving of expensive OT assignment every batch into fitting an GMM once for all, but GMM has been known to suffer the curse of dimensionality, especially the ambient dimension of the image space is very considerable high compared to the number of data points available. Also, the posterior responsibility in high-dimension tends to be nearly one-hot that essentially acts like hard assignments always. I found very little details about the GMM fitting and any conceptual discussions around it for feasibility and practicality. These are legitimate computation overheads of the entire system too, besides the training of network Wϕ\mathbf{W}_\phi.
  • It seems the method requires fitting of GMM on the exact same training data as the released pre-trained flow-matching model, and such training data is not always available despite the model checkpoints are available.
  • The empirical evidences are relatively shallows compared to theoretical contributions. Having latent space flow matching or conditional generation where the conditions come from user utilities such as texts, classes etc, on top of the mode labels that comes from the target distribution. Though η=2τ2\eta=2\tau^2 is clean, but there is no reason not to sweep/ablate on η\eta for empirical confirmation. Plus, the trained network Wϕ\mathbf{W}_\phi seems to absorb multiple things at once, so the improvement brought by the network cannot be credited to the proposed guidance structure specifically without any empirical validation.
Questions for authors

Please try to prioritize the critical weaknesses first, while the remainings are some minor points:

  • It's not clear what the authors try to connect with the time distribution shift in Esser et al. (2024). There, the motivation has something to do with the image resolution (i.e., ambient space dimension), but it seems the paper just wants to emphasize that the shared form falls out of the math, without further implications.
  • Though FID score has been popularly used, a score that is above 30 visually looks bad and has nothing practically usable, and it looks like where the gains are large compared to Gaussian baseline (1-8 Euler steps) exactly has these high FID; thus, if the few-step experiment is to make a point on the straightness, some straightness metrics can also be considered such as in Pooladian et al. (2023).

References

  • Esser et al. (2024) Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
  • Pooladian et al. (2023) Multisample Flow Matching: Straightening Flows with Minibatch Couplings
2 comments

We thank you for the careful review. The changes to the paper are in red in the revised PDF, whose equation and table numbers we use here (a new Eq. (11) and a new Table 1 shift the later numbers by one).

Why a mixture source. Our method does not rely on the target having separated modes. Gaussian mixtures are universal approximators, so a fitted mixture brings the source much closer to the target than a single Gaussian, and the transport cost drops by a factor of 6 to 7 (table below).

Fitting the mixture in high dimension. The fitting procedure is described in Appendix B.2. It takes 16 minutes on CIFAR-10 and 76 minutes on CelebA-HQ on one V100, under 0.15 percent of the training cost of the base network (cost estimates in our reply to Reviewer bUb8). The mixture is not asked to model the images. That is the role of the pretrained network, which supplies most of the velocity, and the mixture only improves it. The responsibilities of clean images are indeed nearly one-hot, which makes the coupling a near hard assignment but this does not change the theory.

Availability of the training data. You are right, the method needs the training data of the pretrained model, both to fit the mixture and to train WϕW_\phi. The data is needed only once, so whoever holds it can release the fitted mixture and WϕW_\phi with the checkpoint, and sampling then requires no data.

Latent and conditional generation. The construction only uses the velocity of the pretrained model and its training data, so it applies to latent and to conditional flow matching models as well.

Sweep of η\eta. The value η=2τ2\eta = 2\tau^2 is not a tuned hyperparameter. It is the temperature at which the duals ψ\psi of Proposition 1 vanish, so that the optimal coupling is the mixture posterior in closed form. For any other η\eta the KK duals have to be solved numerically. The coupling is then no longer the posterior, so the closed-form expressions of Section 4.2 would have to be rederived with the duals and a second scale which is beyond the scope of this paper.

Credit for the improvement of WϕW_\phi. To isolate the guidance structure, we retrained WϕW_\phi without the analytical term of Eq. (20), with the same inputs (table in our reply to Reviewer bUb8). On CIFAR-10 the network alone is slightly better from 4 steps on, but on CelebA-HQ it is worse at every step count. The untrained model, which has no network at all, also brings more than half of the gain over the baseline on CelebA-HQ.

Time shift of Esser et al. Proposition 4 shows that a velocity trained with a standard Gaussian source can be reused with a Gaussian source of scale τ\tau, up to a rescaling of the state and a change of time from tt to s(t)s(t). This change of time has the same form as the timestep shift of Esser et al. (2024). Their motivation is the image resolution, as you note, and they do not mention this fact. We only point out the coincidence.

Straightness. We do not claim straighter trajectories in the paper, but we followed your suggestion. On 1000 trajectories with 256 Euler steps, we measured the path length L=E∫01∥vt(Xt)∥ dtL = \mathbb{E}\int_0^1 \|v_t(X_t)\| \, dt, the transport cost C=E∥X1−X0∥2C = \mathbb{E}\|X_1 - X_0\|^2 and the straightness S=∫01E∥(X1−X0)−vt(Xt)∥2 dtS = \int_0^1 \mathbb{E}\|(X_1 - X_0) - v_t(X_t)\|^2 \, dt of Liu et al. (2023).

path length LLtransport cost CCstraightness SS
CIFAR-10, Gaussian57.93225143.3
CIFAR-10, GMM untrained22.044942.7
CIFAR-10, GMM trained22.446744.4
CelebA-HQ, Gaussian494.32414183447
CelebA-HQ, GMM untrained199.9392021144
CelebA-HQ, GMM trained202.6402171242

On both datasets the coupled paths are about 2.5 times shorter and SS is about 3 times smaller. Relative to their length the coupled paths are not straighter, so the gain comes from transporting each sample over a much shorter distance. The gains also hold at usable quality, for instance 11.24 against 15.00 at 64 steps and 9.80 against 11.45 at 128 steps on CelebA-HQ. We added this table to Appendix B.3 of the revision and a sentence to Section 5.3.

Thank you for your reply. I particularly like the conceptual and theoretical results, and the perspective of EOT and GMM applied to pre-trained flow-based generative models is very fresh to me. Though the known prior results are there, you don't see such an epiphany everyday. However, the empirical results are thin in coverage, and practicality relies on the released training data alongside the pre-trained models. My overall assessment remains positive: weak accept.

Reviewer_WqmMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors replace the Gaussian source in standard flow matching models with a Gaussian mixture fit to the target distribution, and couple each data point to a mixture component through the GMM posterior. They show the resulting velocity equals the pretrained Gaussian-source velocity evaluated at a shifted time, plus a correction that is mostly closed form. This enables improving existing CIFAR-10 and CelebA-HQ checkpoints trained with Gaussian-source FM without retraining, and sampling them with fewer NFE. The authors show that the closed-form approach achieves the largest gain, while a small learned correction further slightly improves performance.

Critical strengths
  • The paper proposes a GMM source coupling strategy that improves existing FM baselines without retraining the FM model, and validates the approach on CIFAR-10 and CelebA-HQ. The strategy is theoretically supported, and further improves generation quality with up to half the NFE of pretrained Gaussian-source FM methods.
  • The untrained variant (Wϕ=0W_\phi = 0) already captures most of the gain, providing a strong training-free method after GMM fitting. Sec 5.4 further provides evidence that the fitted source does not cause training set memorization.
Critical weaknesses
  • Missing comparisons on established benchmarks such as ImageNet-1K 256x256. The paper only compares against Gaussian-source FM on CIFAR-10 and CelebA-HQ, which are insufficient for understanding the utility of the proposed approach. Comparison on published checkpoints of at least 400K iterations (e.g., SiT-B/XL) on ImageNet would be valuable (e.g., on the null-conditioned SiT model), across varying NFE. This shouldn't require any training beyond fitting the GMM, and the W network.
  • Missing compatibility experiments with CFG: It is unclear how the method interfaces with other approaches that encourage early mode commitment such as CFG.
  • The method is positioned as a retraining-free benefit, yet it is unclear how much of a gap to the retraining baseline it closes: How does training a FM model from scratch on the GMM source distribution with the posterior coupling of Eq. 8 conditioned on k, at the same compute as the Gaussian-source FM model on CIFAR-10, compare to the GMM coupled trained/untrained results reported?
Questions for authors

These are further experiment questions, though not under "critical weaknesses":

  • Only Euler sampling is shown. The untrained variant of the method trails the Gaussian source FM approach at NFE=32/64 on CIFAR-10. How does the approach behave compared to the baseline on second-order samplers such as Heun?
  • K is selected on the test-set for CIFAR-10. K should be selected on a subset of the train set. Table 5 suggests it is fairly robust to K, but it would be better for the submission if selection is done on this subset instead. Otherwise, it is unclear how much the improvement over the baseline is due to the advantage of test-set selection.
2 comments

We thank you for the careful review. The changes to the paper are in red in the revised PDF, whose equation and table numbers we use here (a new Eq. (11) and a new Table 1 shift the later numbers by one).

ImageNet-1K 256. We applied the untrained model to the released SiT-XL/2 checkpoint (Ma et al., 2024), in the latent space of its autoencoder, with the null label and no classifier-free guidance. The mixture has K=4096K = 4096 components and is fitted by maximum likelihood to the 1.281.28M training latents and their flips.

FID at Euler steps1248163264128256
SiT-XL/2, Gaussian447.91244.94148.4988.7261.7952.8548.4747.2546.04
SiT-XL/2, GMM untrained269.60196.80119.5373.3056.3150.1147.9346.4846.12

The coupled model is ahead at every step count up to 128128 and ties at 256256. The closed-form guidance therefore transfers to the latent space of a large published model with no training at all. We did not have the time to train the correction network at the data coverage of the paper. Its 100100k steps at batch size 1616 cover the 2.562.56M latents 0.60.6 times, against 1616 and 5353 times for CIFAR-10 and CelebA-HQ, and matching that would take 1010 to 3333 days on one GPU.

Classifier-free guidance. Several mechanisms of our method are also used in conditional sampling, starting with the guidance field of Eq. (12), which steers the base velocity toward a mixture component. Despite these similarities, the two methods have different goals. Classifier-free guidance targets the conditional distribution p(x∣c)p(x \mid c) of a class or prompt cc, and needs labeled data to train its conditional model. Our method samples the unconditional distribution p(x)p(x) and needs no labels, since its components come from the fitted mixture. The two goals are not contradictory and the methods can be combined.

Retraining baseline. We cannot run the baseline you describe within the discussion period. Training from scratch takes about 207 V100-hours on CIFAR-10, and even longer on CelebA-HQ, about 4500 V100-hours (cost estimates in our reply to Reviewer bUb8). We therefore cannot say how much of the gap to retraining our method closes. Such a cost is exactly what our method avoids, since it only adds 1.7 hours of fitting and training to the released model on CIFAR-10, under one percent of that budget. Retraining under a mixture source is the setting of prior work (Jia et al., 2024, Issachar et al., 2025, Luo et al., 2026), whereas our method keeps the pretrained network frozen.

Second-order sampler. We ran Heun on both datasets with the models and the protocol of Table 3, and compare at equal numbers of network evaluations (NFE), up to the budget of the 256-step Euler runs.

FID with Heun, at NFE137153163127255
CIFAR-10, Gaussian371.16102.0419.077.875.534.033.312.94
CIFAR-10, GMM untrained151.0745.8214.008.175.634.053.172.88
CIFAR-10, GMM trained151.6945.8512.497.755.293.773.142.90
CelebA-HQ, Gaussian258.12153.6997.8953.9927.3014.1810.049.29
CelebA-HQ, GMM untrained187.62104.8865.0735.8618.6611.649.519.31
CelebA-HQ, GMM trained182.5290.3056.9231.9516.6510.279.189.16

On CIFAR-10 the trained model is ahead of the baseline at every budget. The untrained model, which trailed at 32 and 64 Euler steps, is within 0.3 of the baseline from 15 to 63 evaluations and ahead elsewhere. On CelebA-HQ the trained model is ahead at every budget, and the untrained model is ahead up to 127 evaluations and within 0.02 of the baseline at 255. We added this table to the revision (Table 4) and a sentence to Section 5.3.

Selection of KK without the test set. We split the CIFAR-10 training set at random into 45k images for the fit and 5k held-out images, refitted the mixture for every KK of the sweep with the same settings, and measured its log-likelihood per dimension on the held-out images. It is −0.418-0.418, −0.370-0.370, −0.337-0.337, −0.326-0.326 and −0.334-0.334 for K=64K = 64, 256256, 10241024, 40964096 and 1638416384. This criterion therefore selects K=4096K = 4096, the value we had chosen on the test set. On CelebA-HQ, where we chose KK by images per component, the same criterion with 3k held-out images gives −0.615-0.615, −0.525-0.525, −0.463-0.463, −0.417-0.417, −0.395-0.395 and −0.392-0.392 for K=4K = 4 to 40964096, so it still slightly improves at K=4096K = 4096 and our K=1024K = 1024 is slightly conservative.

Reviewer_WqmMOpenReview ↗

Response to authors

I thank the authors for their response.

The untrained GMM results are promising on ImageNet, improving FID at nearly every step (though the gains become small by 64 and 128 steps). Additionally, the Heun results are good -- the trained variant outperforms the baseline across NFE, and it is also great that selecting K on a heldout val set matches that used in CIFAR-10.

There are still two points left unresolved:

  1. The authors do not report the trained GMM variant on ImageNet, making it difficult to understand the full utility of their approach on larger scale benchmarks.
  2. The authors state that CFG can be incorporated, yet do not report any results with it in this rebuttal.

Overall however, my perspective has shifted positive as a result of the ImageNet, K selection and Heun results. For the camera ready, I would expect to see a GMM trained result on ImageNet, and a CFG result -- or a statement that the impact of CFG on their approach is untested / unknown, even if the two can be combined in principle.

Reviewer_qGhMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a method to improve pretrained Gaussian-source flows using a fitted Gaussian mixture model (GMM). The authors first fit a GMM and use its component responsibilities to define conditional target distributions. They then show that the velocity field from a standard Gaussian to each conditional target can be expressed using the pretrained velocity and a classifier guidance term (Equation 11). Proposition 4 further relates this conditional velocity to the one transporting the corresponding Gaussian component to its conditional target through an affine transformation and time reparameterization.

The required classifier is unknown, but Proposition 5 expresses it using an analytical term and an unknown function FtF_t. The authors drop log⁡Ft\log F_t from the classifier and propose two variants: an untrained one that uses only the analytical correction, and a trained one that learns an additional correction WϕW_\phi. The methods are evaluated on synthetic 2D data and image-generation benchmarks.

Critical strengths
  1. The approach is interesting and combines several sequential ideas that I consider high quality, particularly in the theoretical development.
  2. The method shows improvements in several settings, even when only fitting the GMM without training an additional correction network.
  3. I particularly liked Equation (11) and Proposition 4, which provide elegant relationships between the different velocity fields.
Critical weaknesses
  1. The presentation could be significantly improved. The paper contains several interesting ideas and results, but unfortunately, I found the manuscript difficult to follow.

  2. Important experimental comparisons are missing. The paper does not compare against a flow trained end-to-end from the same GMM source to the target, nor against a model that directly learns the classifier qt(k∣z)q_t(k\mid z). Since the trained variant already introduces an additional network, these comparisons seem important for establishing the significance of the proposed approach.

  3. I find the treatment of the unknown function FtF_t in Equation (19) questionable. The authors drop log⁡Ft\log F_t from the classifier while learning a correction intended to account for its missing gradients. Consequently, WϕW_\phi does not actually learn these gradients, as it must also compensate for the approximation of the classifier. Why not parameterize log⁡Ft\log F_t directly and use both the potential and its gradients in the flow-matching loss? The authors argue that learning a potential does not necessarily imply learning its gradients, but this argument does not address the proposed alternative, where the gradients themselves participate in the loss.

  4. The empirical improvements are somewhat underwhelming and dataset-dependent, particularly for the untrained variant. On CIFAR-10, the untrained approach is sometimes worse than the baseline, while the improvements of the trained variant become relatively small at higher step counts. The CelebA-HQ results are more convincing, but overall, I would have liked to see stronger or more consistent improvements.

Questions for authors

Q1. Why not parameterize log⁡Ft\log F_t, compute its gradients, and use both in the flow-matching loss? This would allow the gradients to participate directly in training.

Q2. How would the proposed approaches compare against a flow trained end-to-end from the same GMM source to the target?

Q3. How would the proposed approaches compare against directly training a classifier qt(k∣z)q_t(k\mid z) and using its gradients for guidance?

Q4. Could the authors report NLL comparisons for the image experiments on both datasets?

1 comment

We thank you for the careful review. The changes to the paper are in red in the revised PDF, whose equation and table numbers we use here (a new Eq. (11) and a new Table 1 shift the later numbers by one).

Presentation. We rewrote several passages of the paper for clarity.

End-to-end training from the mixture source. We cannot run this baseline within the discussion period, since retraining the base network takes about 207 V100-hours on CIFAR-10 and 4500 on CelebA-HQ (cost estimates in our reply to Reviewer bUb8). It would also not be a fair comparison as it needs more than 100 times the compute of our method.

Learning the classifier directly. We ran this comparison. Eq. (18) gives every term of the logits in closed form except log⁡Ft\log F_t evaluated at the KK points mkm_k. We replaced its KK evaluations by a convolutional network that maps the state zz and the time tt to KK values, which enter the softmax alongside the closed-form terms. We trained it by cross-entropy on the labels of the coupling, selected it on held-out images, and used the gradient of the resulting classifier in Eq. (12) with the frozen base network.

CIFAR-10, FID at Euler steps1248163264128256
K=4096K = 4096, untrained (Table 3)151.4385.6231.3614.498.485.654.143.322.98
K=4096K = 4096, learned classifier gradient151.2685.2831.1814.538.515.694.043.383.02
K=64K = 64, untrained (Table 6)312.40120.0036.3514.818.395.574.023.382.98
K=64K = 64, learned classifier gradient310.52119.1336.1214.888.525.574.203.443.09

The learned classifier does not improve on the closed-form posterior. With K=4096K = 4096 a component holds only 12 distinct training images on average, and the network overfits them and never beats the closed form in held-out cross-entropy. We therefore repeated the comparison with K=64K = 64, where a component holds about 780 images. There the network does generalize, but its gradient still brings no gain.

Parameterizing log⁡Ft\log F_t directly. This is a natural alternative, but it would be much costlier than our parameterization. The posterior and the correction of Eq. (19) involve log⁡Ft\log F_t and its gradient at the KK points mjm_j, so a network for log⁡Ft\log F_t needs KK forward and backward passes per sample and per sampling step. Training with the gradient in the loss adds a second backpropagation through these KK evaluations. Our network WϕW_\phi avoids this with a single forward pass per step.

Size and consistency of the improvements. These improvements are almost free. The base network stays frozen, the mixture fit and the correction network together cost under 1 percent of its training (cost estimates in our reply to Reviewer bUb8), and a sampling step takes 1 to 5 percent longer. For this price, the trained model improves FID at every step count on both datasets and halves the number of steps on CelebA-HQ.

Negative log-likelihood. We computed it on the 10k CIFAR-10 test images with the probability flow ODE (RK45, one Hutchinson probe per image). For a coupled model we integrate the flow of the image's own component, which gives an upper bound, loose by at most log⁡2K/d=0.004\log_2 K / d = 0.004 bits per dimension for an exact flow. The coupling leaves the likelihood almost unchanged, at 3.163 for the trained model and 3.164 for the untrained one against 3.152 for the Gaussian baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.