- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a dynamic classifier-free guidance (CFG) method based on adversarial learning. Unlike CFG methods that rely on fixed weights or manually designed schedules, it models the guidance weight as a function of time, the text condition, and the current noisy sample state, adapting to the needs of different prompts and generation stages. The method introduces a guidance network and a discriminator, using adversarial training to reduce the discrepancy between the marginal distributions of guided samples and real samples with added noise. It also incorporates a CLIP reward to improve text alignment. During inference, the guidance network dynamically adjusts the CFG strength at each step. In MS-COCO text-to-image experiments with two sizes of MMDiT models, the method improves text alignment, aesthetics, and some human preference metrics, but worsens FID.
Critical strengths
(1) Relaxing per-pair consistency in favor of distribution-level adversarial matching. The paper replaces the MMD-based marginal consistency constraint in Learn to Guide with adversarial density-ratio estimation, using a learned distribution-matching signal to relax consistency requirements on individual image–text pairs, and conditions on the current generation state to learn finer-grained CFG schedules. Per-pair consistency is thereby relaxed and matching is performed at the distribution level, giving the guidance schedule more room for optimization and directly addressing the overly tight consistency constraint and limited scheduling flexibility of the original framework.
(2) Concrete evidence at two model scales within a single architecture. Experiments cover two model sizes of the same MMDiT architecture, ablate design choices such as state input and time sampling, and report improvements in text alignment, aesthetics, and some preference metrics. For my assessment, this moves the method from design motivation to concrete evidence: the evaluation spans two model scales and the gains appear across multiple metrics, which is the practical reason I support acceptance.
Critical weaknesses
The current experiments cover only two model sizes within the same MMDiT architecture, which is insufficient to fully establish the method’s applicability to different pretrained generative models. I recommend training separate guidance networks on additional publicly available models that support CFG and comparing them against fixed CFG, limited-interval guidance, and Learn to Guide. FID, text alignment, and preference reward metrics should all be reported to assess whether the observed gains and quality trade-offs can be reproduced across models.
Questions for authors
-
Applicability across backbones: The current experiments cover only two model sizes from the same MMDiT family. Although the authors state in the limitations section that the scheduler must be retrained when the backbone changes, they have not established whether such retraining yields consistent gains on other models. Could the authors conduct additional experiments on other publicly available pretrained models that support CFG, train a separate guidance network for each, and report FID, text alignment, and preference metrics relative to fixed CFG, limited-interval guidance, and Learn to Guide to establish the method’s applicability across models? Examples: FLUX 3 Image Klein Base GitHub: https://github.com/black-forest-labs/flux-3-image-klein Weights: https://huggingface.co/black-forest-labs/flux-3-image-klein-base
-
Training resources and cost: Appendix D and Table 16 report the number of training steps, batch size, and optimizer settings, but do not specify the GPU model, number of GPUs, or actual training time. Could the authors provide the wall-clock time and GPU-hours required to jointly train the guidance network and discriminator for each of the two backbones, so that the reproducibility cost and practicality of the method can be assessed?
3 comments
I appreciate the new public-backbone experiments and the detailed compute table, which substantially address my question about training cost. I also accept SD3.5-Medium and SD1.5 as reasonable substitutes for the example backbones.
Nevertheless, the requested Learn to Guide comparison is still absent. LIG and CLG are not Learn to Guide, i.e., MMD+SC/MMD+MC, and this is the closest baseline for the claimed contribution. More importantly, the new results show a trade-off rather than consistent improvement: on SD3.5-Medium, the proposed method has higher FID than tuned fixed guidance, LIG, and CLG, while its gains in CLIP and Aesthetic are marginal. On SD1.5, it remains worse in FID than fixed CFG 2.0 and CLG. The small metric differences are also not supported by seed variance or confidence intervals.
I therefore find the practical benefit and cross-backbone consistency unresolved and keep my rating at 2: weak rejection.
Thank you for confirming that our SD3.5-Medium and SD v1.5 experiments and compute breakdown address your original questions regarding public backbones and training cost. We would like to briefly clarify the remaining three points:
-
Comparison to Learn to Guide (MMD+SC / MMD+MC): Due to the limited rebuttal window we prioritized for training GAN+MC from scratch and running extensive
Fixed CFG,LIG, andCLGsweeps across two new public backbones, alongside multi-step trajectory evaluations. We did not have sufficient time to additionally train and tune MMD+SC/MMD+MC on the new public backbones. However, Learn to Guide is not absent from our evaluation: Tables 1, 2, 5, and 6 of the paper already compare GAN+MC head-to-head against six variants of MMD+SC and MMD+MC across two backbones (MMDiT-XS 740M and MMDiT-S 1.05B) under identical generator architectures and training budgets:- MMDiT-XS (Tables 1 & 5):
GAN+MC(CLIP = 0.3048,Aesthetic = 5.32,HPSv2 = 0.2829,PickScore = 0.2186) outperforms allMMD+SC(0.3039–0.3042CLIP,5.25–5.26Aesthetic,0.2800–0.2813HPSv2) andMMD+MC(0.3034–0.3037CLIP,5.23–5.25Aesthetic,0.2799–0.2807HPSv2) variants across every alignment and preference metric.GAN+MCalso achieves lower FID thanMMD+SC(28.38) andMMD+MC(28.11). - MMDiT-S (Tables 2 & 6):
GAN+MC(CLIP = 0.3068,Aesthetic = 5.34,HPSv2 = 0.2856,PickScore = 0.2214) again beats allMMD+SC(0.3060–0.3067CLIP,5.27–5.29Aesthetic,0.2835–0.2847HPSv2) andMMD+MC(0.3062–0.3063CLIP,5.26–5.28Aesthetic,0.2832–0.2836HPSv2) baselines, whileGAN+MC(s,t,c) (Table 3:FID = 28.00) achieves lower FID than bothMMD+SC(28.55) andMMD+MC(29.43).
- MMDiT-XS (Tables 1 & 5):
-
Inherent FID vs. Alignment Trade-off on SD v1.5 and SD3.5-Medium: In classifier-free guidance, there is a fundamental Pareto trade-off between distributional FID and prompt alignment/perceptual quality and simultaneously achieving the weakly-guided FID minimum and peak text alignment is not possible:
- On SD v1.5 (Global Response table),
Fixed CFG = 2.00(29.46FID) andCLG(31.72–33.64FID) achieve lower FID only by sacrificing alignment and perceptual quality (Fixed CFG = 2.00drops to0.2956CLIP and5.1353Aesthetic, producing weakly aligned images not used in practice. In addition,CLGachieves at0.3024–0.3031CLIP and5.3656–5.4111Aesthetic).
- On SD v1.5 (Global Response table),
-
Multi-Seed Variance (from Our Response to Reviewer
DnuF): We performed checkpoint evaluation across three sampling seeds (0,1,2) on MMDiT-S. These are the numbers we obtained. FID: , CLIP: , Aesthetic: , HPSv2: , PickScore: . We observe small standard deviations in our evaluation above.
We hope this clarifies the Learn to Guide comparisons, the inherent FID-alignment Pareto trade-off, and the statistical significance of our results.
Response to Reviewer VKYb
We thank the reviewer for the encouraging review and for noting under Critical Strengths that our multi-scale gains are "the practical reason I support acceptance." We address both of your requests below.
1. Applicability Across Public Backbones
We trained guidance networks with our method on two widely used open-weight backbones spanning both architectures (MMDiT and UNet) and generative parameterizations (continuous-time v-prediction flow matching and discrete-time ϵ-prediction DDPM/DDIM diffusion):
We summarize representative baselines below (full fine-grained CFG and schedule sweeps are reported in the Global Response).
SD3.5-Medium (2.5B) on 10K MS-COCO Prompts, 40 Sampling Steps
SD v1.5 (860M) on 3K Paired MS-COCO Prompts, 50 DDIM Sampling Steps
Across both public backbones, GAN+MC transfers cleanly across architectures and diffusion/flow parameterizations, achieving the highest CLIP and Aesthetic scores on both models (and highest HPSv2 on SD v1.5, outperforming Fixed CFG =5.25–9.00 and LIG [5,45],w=7.5 on both FID and alignment/preference metrics).
2. Details of Computational Requirements
We report exact accelerator hardware, batch sizes, step times, wall-clock hours, and total trainer chip-hours for every backbone, and will include this table in the revised paper.
We hope that we have adequately answered your questions, and we are happy to answer any follow-up questions.