acceptodds
Under review as a conference paper at ICLR 2027

Adversarial Learning of Classifier-Free Guidance Schedules

Abstract

Modern text-to-image diffusion models rely on classifier-free guidance (CFG) to achieve high image fidelity and text alignment. However, CFG typically applies a static, global scale across all timesteps, samples, and conditions — a choice that is generally suboptimal and can introduce artifacts, as different states may benefit from different levels of guidance. While time-varying schedules are known to improve quality, designing them by hand is non-trivial and application-dependent. In this paper, we learn the guidance schedule as a function of diffusion time, conditioning and the current noisy sample, in order to better align sampled images with the text prompt. We frame this as a density ratio estimation problem: a discriminator is trained to estimate the time-dependent log-density ratio between the true and guided marginal distributions, while a lightweight generator network predicts the optimal, state-dependent guidance scale. Empirically, our approach outperforms both heuristic CFG schedules and prior methods for learning dynamic guidance on text-to-image generation benchmarks.

Primary area:
generative models
Keywords:
classifier-free guidance, flow matching models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VKYbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a dynamic classifier-free guidance (CFG) method based on adversarial learning. Unlike CFG methods that rely on fixed weights or manually designed schedules, it models the guidance weight as a function of time, the text condition, and the current noisy sample state, adapting to the needs of different prompts and generation stages. The method introduces a guidance network and a discriminator, using adversarial training to reduce the discrepancy between the marginal distributions of guided samples and real samples with added noise. It also incorporates a CLIP reward to improve text alignment. During inference, the guidance network dynamically adjusts the CFG strength at each step. In MS-COCO text-to-image experiments with two sizes of MMDiT models, the method improves text alignment, aesthetics, and some human preference metrics, but worsens FID.

Critical strengths

(1) Relaxing per-pair consistency in favor of distribution-level adversarial matching. The paper replaces the MMD-based marginal consistency constraint in Learn to Guide with adversarial density-ratio estimation, using a learned distribution-matching signal to relax consistency requirements on individual image–text pairs, and conditions on the current generation state to learn finer-grained CFG schedules. Per-pair consistency is thereby relaxed and matching is performed at the distribution level, giving the guidance schedule more room for optimization and directly addressing the overly tight consistency constraint and limited scheduling flexibility of the original framework.

(2) Concrete evidence at two model scales within a single architecture. Experiments cover two model sizes of the same MMDiT architecture, ablate design choices such as state input and time sampling, and report improvements in text alignment, aesthetics, and some preference metrics. For my assessment, this moves the method from design motivation to concrete evidence: the evaluation spans two model scales and the gains appear across multiple metrics, which is the practical reason I support acceptance.

Critical weaknesses

The current experiments cover only two model sizes within the same MMDiT architecture, which is insufficient to fully establish the method’s applicability to different pretrained generative models. I recommend training separate guidance networks on additional publicly available models that support CFG and comparing them against fixed CFG, limited-interval guidance, and Learn to Guide. FID, text alignment, and preference reward metrics should all be reported to assess whether the observed gains and quality trade-offs can be reproduced across models.

Questions for authors
  1. Applicability across backbones: The current experiments cover only two model sizes from the same MMDiT family. Although the authors state in the limitations section that the scheduler must be retrained when the backbone changes, they have not established whether such retraining yields consistent gains on other models. Could the authors conduct additional experiments on other publicly available pretrained models that support CFG, train a separate guidance network for each, and report FID, text alignment, and preference metrics relative to fixed CFG, limited-interval guidance, and Learn to Guide to establish the method’s applicability across models? Examples: FLUX 3 Image Klein Base GitHub: https://github.com/black-forest-labs/flux-3-image-klein Weights: https://huggingface.co/black-forest-labs/flux-3-image-klein-base

  2. Training resources and cost: Appendix D and Table 16 report the number of training steps, batch size, and optimizer settings, but do not specify the GPU model, number of GPUs, or actual training time. Could the authors provide the wall-clock time and GPU-hours required to jointly train the guidance network and discriminator for each of the two backbones, so that the reproducibility cost and practicality of the method can be assessed?

3 comments
AuthorsOpenReview ↗

Response to Reviewer VKYb

We thank the reviewer for the encouraging review and for noting under Critical Strengths that our multi-scale gains are "the practical reason I support acceptance." We address both of your requests below.

1. Applicability Across Public Backbones

We trained guidance networks with our method on two widely used open-weight backbones spanning both architectures (MMDiT and UNet) and generative parameterizations (continuous-time vv-prediction flow matching and discrete-time ϵ\epsilon-prediction DDPM/DDIM diffusion):

  1. Stable Diffusion 3.5 Medium (2.5B parameters), a continuous-time flow-matching MMDiT model, evaluated on 10,000 MS-COCO prompts (matching the exact 10K protocol of Tables 1 & 2).
  2. Stable Diffusion v1.5 (860M parameters), a discrete-time ϵ\epsilon-prediction DDPM/DDIM latent diffusion UNet, evaluated on 3,000 paired MS-COCO prompts.

We summarize representative baselines below (full fine-grained CFG and schedule sweeps are reported in the Global Response).

SD3.5-Medium (2.5B) on 10K MS-COCO Prompts, 40 Sampling Steps

Method / ScheduleFID ↓\downarrowCLIP-B/32 ↑\uparrowAesthetic ↑\uparrowHPSv2 ↑\uparrowPickScore ↑\uparrow
Unguided24.010.29205.05860.25970.2081
Constant w=1.5w=1.516.320.30855.15130.27060.2166
Constant w=4.5w=4.5 (SD3.5 default)25.840.31805.34420.28570.2256
Constant w=5.5w=5.527.340.31825.36360.28700.2260
LIG ([8, 36], w=4.5w=4.5)20.600.31535.31880.28130.2231
LIG (`[4, 36], w=4.5w=4.5)23.000.31675.34510.28360.2246
CLG (ω=4.5\omega=4.5, min 1.01.0)22.950.31725.32540.28530.2250
GAN+MC (Ours)27.290.31835.37250.28580.2256

SD v1.5 (860M) on 3K Paired MS-COCO Prompts, 50 DDIM Sampling Steps

Method / ScheduleFID ↓\downarrowCLIP-B/32 ↑\uparrowAesthetic ↑\uparrowHPSv2 ↑\uparrowPickScore ↑\uparrow
Unguided47.240.27034.8880.25390.1999
Fixed CFG =2.00= 2.0029.460.29565.13530.26880.2106
Fixed CFG =5.25= 5.2536.480.30345.37690.27930.2172
Fixed CFG =7.50= 7.50 (SD1.5 default)39.420.30445.41840.28110.2180
Fixed CFG =9.00= 9.0040.780.30435.43980.28160.2182
LIG ([5, 45], w=7.5w=7.5)36.880.30405.42270.27990.2172
CLG (w=5.5w=5.5, a1=2.0a_1 = 2.0)33.640.30315.41110.27950.2172
GAN+MC (Ours)35.330.30465.49540.28240.2173

Across both public backbones, GAN+MC transfers cleanly across architectures and diffusion/flow parameterizations, achieving the highest CLIP and Aesthetic scores on both models (and highest HPSv2 on SD v1.5, outperforming Fixed CFG =5.25= 5.25–9.009.00 and LIG [5,45],w=7.5[5, 45], w=7.5 on both FID and alignment/preference metrics).


2. Details of Computational Requirements

We report exact accelerator hardware, batch sizes, step times, wall-clock hours, and total trainer chip-hours for every backbone, and will include this table in the revised paper.

BackboneDenoiser ParamsGuidance Net ωψ\omega_{\psi} ParamsTrainer HardwareBatch SizeStep TimeFull Training BudgetInference Overhead (FLOPs / Latency)
MMDiT-XS741M2.90M (0.39%)16×16 \times TPU v5p1280.60 s10.0–10.7 h (160–171 chip-h)<0.001< 0.001\\% / <0.2< 0.2\\%
SD v1.5 (ϵ\epsilon-pred DDIM UNet)860M2.90M (0.34%)16×16 \times TPU v5p1280.91 s15.2 h (243 chip-h)<0.001< 0.001\\% / <0.2< 0.2\\%
MMDiT-S1.05B2.90M (0.28%)16×16 \times TPU v5p1281.14 s18.9–19.4 h (302–310 chip-h)<0.001< 0.001\\% / ≈0.2\approx 0.2\\%
SD3.5-Medium (vv-pred MMDiT)2.5B3.55M (0.16%)16×16 \times TPU v5p1281.05 s16.9–17.5 h (270–280 chip-h)<0.0001< 0.0001\\% / <0.2< 0.2\\%

We hope that we have adequately answered your questions, and we are happy to answer any follow-up questions.

I appreciate the new public-backbone experiments and the detailed compute table, which substantially address my question about training cost. I also accept SD3.5-Medium and SD1.5 as reasonable substitutes for the example backbones.

Nevertheless, the requested Learn to Guide comparison is still absent. LIG and CLG are not Learn to Guide, i.e., MMD+SC/MMD+MC, and this is the closest baseline for the claimed contribution. More importantly, the new results show a trade-off rather than consistent improvement: on SD3.5-Medium, the proposed method has higher FID than tuned fixed guidance, LIG, and CLG, while its gains in CLIP and Aesthetic are marginal. On SD1.5, it remains worse in FID than fixed CFG 2.0 and CLG. The small metric differences are also not supported by seed variance or confidence intervals.

I therefore find the practical benefit and cross-backbone consistency unresolved and keep my rating at 2: weak rejection.

Thank you for confirming that our SD3.5-Medium and SD v1.5 experiments and compute breakdown address your original questions regarding public backbones and training cost. We would like to briefly clarify the remaining three points:

  1. Comparison to Learn to Guide (MMD+SC / MMD+MC): Due to the limited rebuttal window we prioritized for training GAN+MC from scratch and running extensive Fixed CFG, LIG, and CLG sweeps across two new public backbones, alongside multi-step trajectory evaluations. We did not have sufficient time to additionally train and tune MMD+SC/MMD+MC on the new public backbones. However, Learn to Guide is not absent from our evaluation: Tables 1, 2, 5, and 6 of the paper already compare GAN+MC head-to-head against six variants of MMD+SC and MMD+MC across two backbones (MMDiT-XS 740M and MMDiT-S 1.05B) under identical generator architectures and training budgets:

    • MMDiT-XS (Tables 1 & 5): GAN+MC (CLIP = 0.3048, Aesthetic = 5.32, HPSv2 = 0.2829, PickScore = 0.2186) outperforms all MMD+SC (0.3039–0.3042 CLIP, 5.25–5.26 Aesthetic, 0.2800–0.2813 HPSv2) and MMD+MC (0.3034–0.3037 CLIP, 5.23–5.25 Aesthetic, 0.2799–0.2807 HPSv2) variants across every alignment and preference metric. GAN+MC also achieves lower FID than MMD+SC (28.38) and MMD+MC (28.11).
    • MMDiT-S (Tables 2 & 6): GAN+MC (CLIP = 0.3068, Aesthetic = 5.34, HPSv2 = 0.2856, PickScore = 0.2214) again beats all MMD+SC (0.3060–0.3067 CLIP, 5.27–5.29 Aesthetic, 0.2835–0.2847 HPSv2) and MMD+MC (0.3062–0.3063 CLIP, 5.26–5.28 Aesthetic, 0.2832–0.2836 HPSv2) baselines, while GAN+MC (s,t,c) (Table 3: FID = 28.00) achieves lower FID than both MMD+SC (28.55) and MMD+MC (29.43).
  2. Inherent FID vs. Alignment Trade-off on SD v1.5 and SD3.5-Medium: In classifier-free guidance, there is a fundamental Pareto trade-off between distributional FID and prompt alignment/perceptual quality and simultaneously achieving the weakly-guided FID minimum and peak text alignment is not possible:

    • On SD v1.5 (Global Response table), Fixed CFG = 2.00 (29.46 FID) and CLG (31.72–33.64 FID) achieve lower FID only by sacrificing alignment and perceptual quality (Fixed CFG = 2.00 drops to 0.2956 CLIP and 5.1353 Aesthetic, producing weakly aligned images not used in practice. In addition, CLG achieves at 0.3024–0.3031 CLIP and 5.3656–5.4111 Aesthetic).
  3. Multi-Seed Variance (from Our Response to Reviewer DnuF): We performed checkpoint evaluation across three sampling seeds (0,1,2) on MMDiT-S. These are the numbers we obtained. FID: 31.43±0.151(31.28,31.44,31.58)31.43 \pm 0.151 (31.28, 31.44, 31.58), CLIP: 0.30676±0.000069(0.30680,0.30668,0.30679)0.30676 \pm 0.000069 (0.30680, 0.30668, 0.30679), Aesthetic: 5.3263±0.00272(5.3243,5.3251,5.3294)5.3263 \pm 0.00272 (5.3243, 5.3251, 5.3294), HPSv2: 0.28563±0.000047(0.28561,0.28568,0.28559)0.28563 \pm 0.000047 (0.28561, 0.28568, 0.28559), PickScore: 0.22156±0.000023(0.22154,0.22158,0.22156)0.22156 \pm 0.000023 (0.22154, 0.22158, 0.22156). We observe small standard deviations in our evaluation above.

We hope this clarifies the Learn to Guide comparisons, the inherent FID-alignment Pareto trade-off, and the statistical significance of our results.

Reviewer_DnuFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes to learn classifier-free guidance weights as a function of time, conditioning, and the current noisy sample. The learning problem is then framed as a density ratio estimation problem and learned via GAN: a discriminator is trained to estimate the time-dependent log-density ratio between the true and guided marginal distributions, and a lightweight generator network is trained to predict the current guidance weights. Experiments on two MMDiT model sizes show improvements in alignment and preference scores but with worse FID.

Critical strengths
  • The guidance framework with personalized weights is reasonable, and once the training is completed, the inference stage only adds a lightweight guidance network to make the sampling efficient.
  • The experiments include both MMD+SC and MMD+MC baselines with the same guidance-network architecture, together with useful ablations. These comparisons help assess the effectiveness of the proposed method as compared with existing consistency-based guidance frameworks.
Critical weaknesses

The experimental advantage is generally small and comes with worse FID. Specifically, the increase in FID is actually significant as compared to the improvement in other metrics like PickScore, Aesthetic, etc. Although the authors discuss this trade-off, they may need to acknowledge the significant increase in FID and discuss more deeply how this would affect the practical usage of the proposed method. Currently it remains unclear whether the proposed method offers a meaningful improvement.

Questions for authors
  • The density-ratio formulation does not seem to fully match the stated discriminator: Eq. (13) conditions on cc, whereas Eqs. (7)–(8) integrate out cc. If each sample is paired with its corresponding cc, the optimal discriminator estimates a conditional log-density ratio. Are Eqs. (10)–(13) intended to use conditional distributions given cc, and if so, could the authors make the definitions and notation consistent?

  • The experimental comparisons need more clarification, as the current trade-off between FID and other metrics does not convincingly demonstrate a meaningful improvement. Maybe the authors can consider either (i) reporting standard deviations across repeated runs to help assess whether the gains in the other metrics are larger than the variability; or (ii) comparing against a sweep of constant CFG weights at similar FID to help establish whether the learned schedule improves the trade-off or mainly achieves higher alignment through stronger guidance.

  • In Figure 3, although the learned guidance weights vary across prompts, they seem to follow similar shapes, with the largest weights generally around the middle 30%-70% of the sampling process. It would be interesting if the authors could comment on this observation (or maybe correct me that this is not a valid observation) and whether searching over a finite set of similarly shaped schedules could reduce the computational overhead.

  • The paper uses GAN for training, with regularization and spectral normalization to improve stability. It would be useful to clarify how sensitive the learned guidance weights and final performance are to training seeds and the relative learning rates of the generator and discriminator.

  • There seems to be a mismatch between Figure 1 and its caption. The caption refers to two independent samples, x11x_1^1 and x12x_1^2, whereas the figure shows xtx_t and xsx_s. Making the labels consistent would help clarify the figure.

Some minor issues I noticed:

  • In loss eq(20), it would be better to make the hyperparameter aa explicit in front of R(w,t)R(w,t) to clearly indicate that the three parts in the loss are combined by flexible weights.
  • Missing a period in eq(15).
3 comments
AuthorsOpenReview ↗

Response to Reviewer DnuF

We thank the reviewer for the careful, constructive review, the positive soundness rating, and the mathematical and empirical suggestions. We address your questions and concerns below.

1. Notation on Conditional Distributions (Eqs. 10–13)

Thank you for catching this notational inconsistency. Eqs. (10)–(13) are indeed intended to condition on cc, and we will update Eqs. (10)–(13) to use conditional distributions consistently in the revised manuscript.


2. Constant CFG Sweeps at Similar FID

As requested, we compare against sweeps of constant CFG weights at similar FID on MMDiT-S and SD v1.5 below (full baseline sweeps including SD3.5-Medium are also reported in the Global Response). At matched or lower FID, we observe consistent modest gains in alignment and preference metrics compared to constant CFG.

MMDiT-S (1.05B) on 10K MS-COCO Prompts, 128 Sampling Steps

Method / Guidance WeightFID ↓\downarrowCLIP ↑\uparrowAesthetic ↑\uparrowHPSv2 ↑\uparrowPickScore ↑\uparrow
Unguided24.30.2794.840.25880.2024
Constant ω=2.5\omega = 2.525.020.30425.180.28030.2187
Constant ω=5.0\omega = 5.029.260.30615.270.28360.2208
Constant ω=7.5\omega = 7.531.110.30635.310.28500.2214
Constant ω=10.0\omega = 10.031.790.30655.310.28530.2214
Constant ω=12.5\omega = 12.531.980.30655.320.28650.2213
GAN+MC 128 steps (64 steps)31.70 (30.55)0.3068 (0.3068)5.34 (5.32)0.2856 (0.2853)0.2214 (0.2212)

SD v1.5 (860M) on 3K Paired MS-COCO Prompts, 50 DDIM Sampling Steps

Method / ScheduleFID ↓\downarrowCLIP-B/32 ↑\uparrowAesthetic ↑\uparrowHPSv2 ↑\uparrowPickScore ↑\uparrow
Unguided47.240.27034.8880.25390.1999
Fixed CFG =2.00= 2.0029.460.29565.13530.26880.2106
Fixed CFG =3.00= 3.0031.120.30065.25930.27450.2144
Fixed CFG =4.00= 4.0033.480.30235.33090.27730.2161
Fixed CFG =5.25= 5.2536.480.30345.37690.27930.2172
Fixed CFG =6.00= 6.0037.650.30395.39520.28000.2176
Fixed CFG =7.50= 7.50 (SD1.5 default)39.420.30445.41840.28110.2180
Fixed CFG =9.00= 9.0040.780.30435.43980.28160.2182
GAN+MC (Ours)35.330.30465.49540.28240.2173

In addition, as requested, we performed checkpoint evaluation across three sampling seeds (0,1,2)(0, 1, 2) on MMDiT-S. These are the numbers we obtained. FID: 31.43±0.151(31.28,31.44,31.58)31.43 \pm 0.151 (31.28, 31.44, 31.58), CLIP: 0.30676±0.000069(0.30680,0.30668,0.30679)0.30676 \pm 0.000069 (0.30680, 0.30668, 0.30679), Aesthetic: 5.3263±0.00272(5.3243,5.3251,5.3294)5.3263 \pm 0.00272 (5.3243, 5.3251, 5.3294), HPSv2: 0.28563±0.000047(0.28561,0.28568,0.28559)0.28563 \pm 0.000047 (0.28561, 0.28568, 0.28559), PickScore: 0.22156±0.000023(0.22154,0.22158,0.22156)0.22156 \pm 0.000023 (0.22154, 0.22158, 0.22156)

3. Bell-Shaped Curve of Learned Guidance Schedules

Your observation regarding the shape of the learned schedule is accurate. We believe this shape emerges for three reasons across the trajectory:

  1. Near pure noise, conditional and unconditional velocity fields nearly coincide and SNR→0\text{SNR} \to 0, so a large ω\omega primarily amplifies noise and harms image quality which is consistent with Limited Interval Guidance (Kynkäänniemi et al, 2024).
  2. In the intermediate window (t∈[0.3,0.7]t \in [0.3, 0.7]), global layout and prompt semantics emerge, where the CFG difference term carries the strongest semantic signal.
  3. Close to data, our regularizer and discriminator suppress high-frequency oversaturation, driving ω\omega down again.

We agree that searching over a finite set of similarly shaped parametric schedules could reduce training overhead, and we leave this exploration to future work. At the same time, in our ablation in Table 3 of the manuscript, we observed metric gains from state-dependence, which would be lost under a fixed non-adaptive schedule. Moreover, training is a one-time cost, and inference overhead is negligible (<0.2< 0.2\\% latency; see Global Response) due to the lightweight guidance predictor network.

4. Stability of GAN training and Learning rate Sensitivity

Although our framework uses adversarial training, optimization is lightweight and stable in practice: the MMDiT/UNet backbone remains frozen, the guidance network outputs a scalar, and the discriminator uses spectral normalization and an R1R_1 penalty (applied every 4 steps). As noted in Table 16 (Appendix), we set the generator learning rate to twice that of the discriminator. We observe that the evaluation metrics remain stable even when we scaling both learning rates across a 20×20\times range (0.1×0.1\times to 2.0×2.0\times the default value). We leave an analysis of variation across full training seeds to future work.

(G,D)(G, D) Learning-Rate Scale (MMDiT-XS, 10K COCO)(lrG,lrD)(\text{lr}_G, \text{lr}_D)FID ↓\downarrowCLIP-B/32 ↑\uparrowAesthetic ↑\uparrowHPSv2 ↑\uparrowPickScore ↑\uparrow
0.1×0.1\times LR(2×10−6,;1×10−6)(2\times 10^{-6},\\; 1\times 10^{-6})29.660.30475.31490.28310.2185
1.0×1.0\times LR (Default in paper)(2×10−5,;1×10−5)(2\times 10^{-5},\\; 1\times 10^{-5})31.310.30485.320.28290.2186
2.0×2.0\times LR(4×10−5,;2×10−5)(4\times 10^{-5},\\; 2\times 10^{-5})31.850.30475.32920.28320.2187

5. Figure 1 Labels and Typographical Fixes

Thank you for catching all three details. We will make the Figure 1 caption labels consistent with the figure, make the weighting hyperparameters explicit in Eq. (20), and add the missing period in Eq. (15) in the revised manuscript.

We hope that we have adequately answered your questions, and we are happy to answer any follow-up questions.

Thank you to the authors for the detailed responses and additional experiments, which clarified some of my previous questions. However, after considering the new results, I still find the improvement over existing guidance strategies relatively modest and not sufficiently consistent across settings to establish a clear practical advantage, so I have decided to maintain my original score.

Reviewer_HkSHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an adversarial learning framework for optimizing dynamic CFG schedules in text-to-image generation. Instead of using fixed or manually designed guidance scales, the proposed method learns conditions-dependent guidance weights through a lightweight network. Specifically, the authors formulate guidance learning as a marginal distribution matching problem, where a discriminator estimates the density ratio between real and guided distributions to provide training signals for the guidance network. Experiments on two flow-matching models demonstrate improvements in text alignment and human preference metrics compared with conventional CFG schedules and prior learned guidance methods, although with some degradation in FID.

Critical strengths
  • The paper introduces an interesting adversarial learning framework for optimizing dynamic CFG schedules through density ratio estimation. Compared with prior MMD-based approaches, the proposed formulation offers a flexible way to enforce marginal consistency.

  • The proposed method learns multiple conditions (state, timestep and text-condition) guidance weights without modifying the pretrained diffusion backbone. The lightweight guidance network introduces negligible inference overhead and demonstrates modest improvements in text alignment and perceptual metrics.

Critical weaknesses
  • The proposed method optimizes marginal consistency through one-step transitions initialized from forward-noised real data. While exact one-step marginal consistency would theoretically imply consistency across multi-step sampling, it remains unclear whether the approximate learned objective maintains this property when errors accumulate along the generated trajectory. Evaluating intermediate marginal distributions during full sampling trajectories would provide stronger evidence for the effectiveness of the proposed objective.
  • The most significant concern is about the substantial performance degradation. Although the proposed method achieves slightly better CLIP, HPSv2, and PickScore results, these improvements are relatively marginal, while the FID deteriorates substantially compared to almost all baselines. This raises concerns about whether the limited gains in alignment and perceptual quality justify the significant degradation in distributional fidelity, especially given that the proposed method explicitly aims to improve marginal distribution matching. Furthermore, the authors' explanations based on the mode-seeking behavior of reverse KL and the misalignment between FID and human preferences do not sufficiently justify such a substantial degradation in FID.
Questions for authors
  • The core motivation of the proposed method is to enforce marginal consistency between the guided sampling distribution and the real data distribution at different denoising timesteps. I think exploring marginal consistency in CFG is very interesting and well worth pursuing. However, the current experiments primarily focus on the quality of the final generated images, without directly test whether the proposed adversarial objective actually improves marginal consistency along the sampling trajectory. I suggest the authors provide a more thorough analysis by measuring the distributional discrepancy between real and guided samples at different timesteps, using metrics such as MMD or held-out discriminator accuracy. Such experiments would provide stronger evidence that the proposed method achieves its intended objective, rather than merely improving certain final image quality metrics.

  • The other questions please refer to the Weakness Part.

2 comments
AuthorsOpenReview ↗

Response to Reviewer HkSH

We thank the reviewer for the insightful feedback and for highlighting the appeal of enforcing marginal consistency in CFG via adversarial density-ratio estimation. We address your concerns and questions below.

1. Evaluations of Marginal Consistency Along Trajectory Rollouts

Thank you for suggesting a direct test of intermediate marginal distributions along full multi-step sampling trajectories rather than only final decoded images. Following your exact recommendation, we evaluated full multi-step sampling trajectories initialized from pure Gaussian noise x0∼N(0,I)x_0 \sim \mathcal{N}(0, I) on N=2048N=2048 held-out MS-COCO validation pairs (x1,c)(x_1, c) at intermediate timesteps t∈0.2,0.4,0.6,0.8,0.9t \in \\{0.2, 0.4, 0.6, 0.8, 0.9\\}. At each tt, we compared the multi-step rollout sample xttrajx_t^{\text{traj}} (integrated from 0→t0 \to t) against ground-truth forward-noised real data xtreal=αtx1+σtϵx_t^{\text{real}} = \alpha_t x_1 + \sigma_t \epsilon using Latent and Prompt-Conditioned Energy-Distance Kernel (MMD$^2$), Fréchet Latent Distance (FLD), and held-out 3-fold CV Classifier Two-Sample Test (C2ST) accuracy (5050\\% = indistinguishable marginals). The latent energy-distance kernel matches the one used in Galashov et al. (2026), and the prompt-conditioned kernel is its conditional counterpart.

Multi-Step Trajectory Marginal Consistency on MMDiT-S (1.05B)

Method (MMDiT-S)Latent MMD$^2$ (×10−3\times 10^{-3}) ↓\downarrow t=0.2 / 0.4 / 0.6 / 0.8 / 0.9 (Mean)Cond. MMD$^2$ ↓\downarrow t=0.2 / 0.4 / 0.6 / 0.8 / 0.9 (Mean)FLD ↓\downarrow (Mean)Held-Out Cond. C2ST Acc. ↓\downarrow t=0.2 / 0.4 / 0.6 / 0.8 / 0.9 (Mean)
Unguided0.21 / 0.90 / 2.09 / 3.23 / 3.74 (2.03)0.012 / 0.032 / 0.041 / 0.039 / 0.038 (0.032)0.51175.4% / 85.4% / 85.9% / 85.6% / 85.6% (83.6%)
Fixed CFG (ω=10.0\omega=10.0)0.64 / 2.61 / 6.48 / 10.26 / 11.88 (6.38)0.136 / 0.228 / 0.263 / 0.276 / 0.277 (0.236)1.83897.4% / 98.4% / 97.8% / 97.6% / 97.4% (97.7%)
Fixed CFG (ω=7.5\omega=7.5 default)0.56 / 2.27 / 5.50 / 8.64 / 10.00 (5.39)0.126 / 0.222 / 0.260 / 0.272 / 0.275 (0.231)1.68096.3% / 97.5% / 97.1% / 96.9% / 96.8% (96.9%)
Fixed CFG (ω=5.0\omega=5.0)0.46 / 1.87 / 4.40 / 6.84 / 7.91 (4.30)0.100 / 0.185 / 0.217 / 0.228 / 0.229 (0.192)1.42993.7% / 95.9% / 95.7% / 95.5% / 95.5% (95.2%)
MMD+SC0.39 / 1.95 / 4.96 / 8.49 / 9.93 (5.14)0.097 / 0.186 / 0.216 / 0.227 / 0.229 (0.191)1.39894.2% / 97.1% / 97.1% / 96.9% / 96.9% (96.4%)
MMD+MC0.37 / 1.70 / 3.95 / 6.38 / 7.36 (3.95)0.087 / 0.176 / 0.205 / 0.214 / 0.215 (0.179)1.28891.9% / 95.4% / 95.2% / 94.9% / 95.2% (94.5%)
GAN+MC (Ours)0.41 / 1.79 / 4.51 / 7.32 / 8.49 (4.50)0.093 / 0.178 / 0.210 / 0.220 / 0.220 (0.184)1.31293.0% / 96.4% / 96.6% / 96.0% / 96.1% (95.6%)

Overall, the unguided trajectory achieves the lowest discrepancy across all metrics, since any nonzero guidance trades marginal fidelity for stronger prompt alignment. Relative to constant CFG (e.g., default ω=7.5\omega=7.5 and ω=10.0\omega=10.0, as well as ω=5.0\omega=5.0 on conditional MMD$^2$ and FLD), our method (GAN+MC) lowers latent MMD$^2$, conditional MMD$^2$, FLD, and C2ST accuracy while approaching MMD-based baselines that explicitly optimize the evaluation MMD kernel. Moreover, within the MMD family, MMD+MC consistently outperforms MMD+SC, confirming that the marginal-consistency objective itself reduces multi-step rollout drift.

2. Trade-off Between FID vs. CLIP and Human Preference Metrics

We agree that our method results in an increase in FID while improving alignment. In general, guidance trades diversity for alignment across every method, and for heuristic baselines, gains in FID come at the cost of reduced alignment. We report additional CFG and baseline sweeps on our MMDiT-S model below (as well as on SD3.5-Medium and SD v1.5 in the Global Response), where we observe modest gains in CLIP and Aesthetic metrics at matched FID.

MMDiT-S (1.05B) Results on 10K MS-COCO Prompts, 128 Sampling Steps

Method / Guidance WeightFID ↓\downarrowCLIP ↑\uparrowAesthetic ↑\uparrowHPSv2 ↑\uparrowPickScore ↑\uparrow
Unguided24.30.2794.840.25880.2024
Constant ω=2.5\omega = 2.525.020.30425.180.28030.2187
Constant ω=5.0\omega = 5.029.260.30615.270.28360.2208
Constant ω=7.5\omega = 7.531.110.30635.310.28500.2214
Constant ω=10.0\omega = 10.031.790.30655.310.28530.2214
Constant ω=12.5\omega = 12.531.980.30655.320.28650.2213
LIG (ω=15.0,[0.1,0.9]\omega=15.0, [0.1, 0.9])24.950.30425.310.28430.2194
CLG (ω=14.0\omega=14.0)27.140.30555.270.28380.2203
MMD+SC28.550.30675.290.28470.2208
MMD+MC29.430.30625.280.28360.2205
GAN+MC128 steps (64 steps)31.70 (30.55)0.3068 (0.3068)5.34 (5.32)0.2856 (0.2853)0.2214 (0.2212)

We hope that we have adequately answered your questions, and we are happy to answer any follow-up questions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response

We thank the reviewers for their careful and constructive reviews. Three main questions and requests recur: 1) whether the method transfers to public backbones, 2) details of computational requirements, and 3) whether the FID increase in Tables 1–2 reflects a genuine alignment-diversity trade-off. We summarize the new evidence here. We also include details in the individual responses and will upload a revised PDF with these results.

1. Results on Additional Public Backbones

We trained guidance networks with our method on Stable Diffusion 3.5 Medium (2.5B, vv-prediction MMDiT, continuous time) and Stable Diffusion v1.5 (860M, ϵ\epsilon-prediction UNet, discrete time) and evaluated them on MS-COCO prompts against constant CFG sweeps, Limited Interval Guidance (LIG), and Clamped Linear Guidance (CLG).

SD v1.5 (860M) Results on 3K Paired MS-COCO Prompts, 50 DDIM Sampling Steps

Method / ScheduleFID ↓\downarrowCLIP-B/32 ↑\uparrowAesthetic ↑\uparrowHPSv2 ↑\uparrowPickScore ↑\uparrow
Unguided47.240.27034.8880.25390.1999
Fixed CFG =2.00= 2.0029.460.29565.13530.26880.2106
Fixed CFG =3.00= 3.0031.120.30065.25930.27450.2144
Fixed CFG =4.00= 4.0033.480.30235.33090.27730.2161
Fixed CFG =5.25= 5.2536.480.30345.37690.27930.2172
Fixed CFG =6.00= 6.0037.650.30395.39520.28000.2176
Fixed CFG =7.50= 7.50 (SD1.5 default)39.420.30445.41840.28110.2180
Fixed CFG =9.00= 9.0040.780.30435.43980.28160.2182
LIG ([10, 45], w=6.0w=6.0)33.050.30215.37720.27670.2154
LIG ([10, 45], w=7.5w=7.5)34.370.30335.41870.27670.2154
LIG ([5, 45], w=6.0w=6.0)35.250.30345.38830.27860.2165
LIG ([5, 45], w=7.5w=7.5)36.880.30405.42270.27990.2172
CLG (w=4.0w=4.0, a1=1.0a_1 = 1.0)31.720.30245.36560.27700.2158
CLG (w=4.0w=4.0, a1=2.0a_1 = 2.0)32.700.30255.36980.27800.2165
CLG (w=5.5w=5.5, a1=1.0a_1 = 1.0)32.740.30315.40760.27890.2167
CLG (w=5.5w=5.5, a1=2.0a_1 = 2.0)33.640.30315.41110.27950.2172
GAN+MC (Ours)35.330.30465.49540.28240.2173

SD3.5-Medium (2.5B) Results on 10K MS-COCO Prompts, 40 Sampling Steps

Method / ScheduleFID ↓\downarrowCLIP-B/32 ↑\uparrowAesthetic ↑\uparrowHPSv2 ↑\uparrowPickScore ↑\uparrow
Unguided24.010.29205.05860.25970.2081
Constant w=1.5w=1.516.320.30855.15130.27060.2166
Constant w=2.0w=2.017.740.31405.20850.27610.2205
Constant w=2.5w=2.519.870.31635.25320.27950.2226
Constant w=3.0w=3.022.080.31725.28720.28180.2239
Constant w=4.0w=4.024.960.31805.32900.28470.2253
Constant w=4.5w=4.5 (SD3.5 default)25.840.31805.34420.28570.2256
Constant w=5.5w=5.527.340.31825.36360.28700.2260
LIG ([8, 36], w=4.5w=4.5)20.600.31535.31880.28130.2231
LIG ([4, 36], w=4.5w=4.5)23.000.31675.34510.28360.2246
CLG (peak ω=3.5\omega=3.5, min 1.01.0)21.960.31725.30650.28370.2245
CLG (peak ω=4.5\omega=4.5, min 1.01.0)22.950.31725.32540.28530.2250
GAN+MC (Ours)27.290.31835.37250.28580.2256

Across both public models (and our 741M and 1.05B MMDiTs, spanning 4 backbones from 741M to 2.5B), GAN+MC achieves competitive performance on alignment and preference metrics. Specifically, on SD v1.5, we outperform heuristic baselines on alignment and preference-based metrics, even at comparable FID.


2. Details of Computational Requirements

We report exact accelerator hardware, batch sizes, step times, wall-clock hours, and total trainer chip-hours for every backbone. We will also update the paper with these details.

BackboneDenoiser ParamsGuidance Net ωψ\omega_{\psi} ParamsTrainer HardwareBatch SizeStep TimeFull Training BudgetInference Overhead (FLOPs / Latency)
MMDiT-XS741M2.90M (0.39%)16×16 \times TPU v5p1280.60 s10.0–10.7 h (160–171 chip-h)<0.001< 0.001\\% / <0.2< 0.2\\%
SD v1.5 (ϵ\epsilon-pred DDIM UNet)860M2.90M (0.34%)16×16 \times TPU v5p1280.91 s15.2 h (243 chip-h)<0.001< 0.001\\% / <0.2< 0.2\\%
MMDiT-S1.05B2.90M (0.28%)16×16 \times TPU v5p1281.14 s18.9–19.4 h (302–310 chip-h)<0.001< 0.001\\% / ≈0.2\approx 0.2\\%
SD3.5-Medium (vv-pred MMDiT)2.5B3.55M (0.16%)16×16 \times TPU v5p1281.05 s16.9–17.5 h (270–280 chip-h)<0.0001< 0.0001\\% / <0.2< 0.2\\%
AuthorsOpenReview ↗

Global Response - Contd


3. FID and Alignment Trade-off

We agree that our method results in an increase in FID while improving alignment. In general, guidance trades diversity for alignment for every method. For heuristic baselines, gains in FID come at the cost of slightly reduced alignment. In addition, as stated in Section 3 of the submission, the generator objective (Eq. 13) is a reverse KL between the guided and data marginals, so the learned schedule is mode-seeking by construction. Below, we report additional CFG sweeps alongside baseline comparisons on our MMDiT-S model (in addition to SD3.5 and SD v1.5 above), where we observe modest gains in CLIP and Aesthetic metrics at matched FID.

MMDiT-S (1.05B) Results on 10K MS-COCO Prompts, 128 Sampling Steps

Method / Guidance WeightFID ↓\downarrowCLIP ↑\uparrowAesthetic ↑\uparrowHPSv2 ↑\uparrowPickScore ↑\uparrow
Unguided24.30.2794.840.25880.2024
Constant ω=2.5\omega = 2.525.020.30425.180.28030.2187
Constant ω=5.0\omega = 5.029.260.30615.270.28360.2208
Constant ω=7.5\omega = 7.531.110.30635.310.28500.2214
Constant ω=10.0\omega = 10.031.790.30655.310.28530.2214
Constant ω=12.5\omega = 12.531.980.30655.320.28650.2213
LIG (ω=15.0,[0.1,0.9]\omega=15.0, [0.1, 0.9])24.950.30425.310.28430.2194
CLG (ω=14.0\omega=14.0)27.140.30555.270.28380.2203
MMD+SC28.550.30675.290.28470.2208
MMD+MC29.430.30625.280.28360.2205
GAN+MC 128 steps (64 steps)31.70 (30.55)0.3068 (0.3068)5.34 (5.32)0.2856 (0.2853)0.2214 (0.2212)
Area_Chair_MQscOpenReview ↗

Encouraging Reviewer Feedback on Author Responses

Dear Reviewers,

Thank you all for your time and effort in reviewing and responding. Just a gentle reminder to reviewers to take a look at the authors' responses and share any updated thoughts on whether your main concerns have been addressed and, if not, to indicate which points are not well addressed. The author–reviewer discussion period ends on November 18 (AoE), so I would greatly appreciate your input before then. Thank you again for your time and contributions!

Kind Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.