- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper focus on the attribute distribution alignment problem of pretrained unconditional diffusion models in the inference stage, aiming to make the attribute distribution of generated samples matches a target distribution without fine-tuning the generative model. The authors model the reverse diffusion process as an optimal control problem, taking the KL divergence of the batch sample attribute distribution as the cost, and introduce a control regularization term, iteratively optimizing the control variables via Pontryagin's maximum principle and the E-MSA algorithm. In addition, the authors provide theoretical analysis on the algorithm's convergence and attribute distribution alignment. Experiments cover semantic classes control on CIFAR-100 and human-face attributes control on FFHQ. Experimental results show that this method can more effectively match the target attribute distribution compared to the selected baselines, while maintaining good generation quality.
Critical strengths
1.The problem modeling aligns well with the distribution matching. Since attribute distribution matching is essentially a group-level objective, the authors did not simply decompose it into independent per-sample rewards, but instead used batch-level KL divergence as the terminal cost to explicitly couple multiple generated samples, and introduced control regularization to balance distribution alignment and generation quality. This modeling approach is natural and reasonable for this problem.
Critical weaknesses
1.The paper does not sufficiently justify why attribute distribution matching should be addressed by controlling pretrained unconditional diffusion models. For semantic classes distribution matching on CIFAR-100, a more direct and natural solution would be using a class-conditional diffusion model and sample according to the desired class distribution. Similarly, for human face generation, text-to-image models with carefully designed text prompts could be used to control the proportions of different attributes.
2.When restricted to pretrained unconditional diffusion models, a straightforward strategy could be considered: generate a sufficiently large pool of candidate images and select samples using an attribute classifier to match the target distribution. Theoretically, as long as every target class has nonzero probability under the original generative distribution, sufficientl independent samples can provide the required number of images of each class, allowing arbitrary target classes distributions to be matched.
3.The experimental comparisons do not adequately cover closely related methods or simpler alternatives. The authors discuss OC-Flow [1] and DTM [2] in the Related Work section, both of which employ optimal control to steer the sampling trajectories of pretrained generative models. However, the paper does not sufficiently clarify the differences from these existing optimal control frameworks, nor does it provide comparisons under a unified batch-level distribution matching objective. Consequently, it remains unclear whether the contribution primarily amounts to replacing the terminal cost function in an existing optimal control framework with a batch-level distribution matching cost.The experiments lack simple but important baselines, such as generate-then-select mentioned in 2, as well as reference comparisons with conditional generation combined with class-quota sampling. According to the complexity analysis in Appendix B.3, the proposed method requires multiple forward simulations and backward gradient computations. With approximately 10 optimization iterations, its generative-model computation is roughly comparable to that of several dozen standard unconditional sampling runs. This raises a natural question: under the same computational budget, could one simply generate more candidate images and select them according to the desired attribute proportions using the Attribute Oracle, achieving comparable or even better distribution alignment?
4.The paper uses the same Attribute Oracle for generation optimization and attribute distribution evaluation, potentially introducing oracle-specific evaluation bias. Although the appendix includes ablation studies using ResNet and ViT as alternative oracles, it does not provide independent cross-oracle evaluations, such as optimizing generation with ResNet and evaluating the resulting attribute distribution with ViT. The current experiments cannot exclude the possibility that the method primarily exploits the decision boundaries of a particular classifier, rather than achieving reliable alignment of the underlying semantic attribute distribution.
References
[1] Luran Wang, Chaoran Cheng, Yizhen Liao, Yanru Qu, and Ge Liu. Training Free Guided Flow-Matching with Optimal Control. International Conference on Learning Representations (ICLR), 2025.
[2] Kushagra Pandey, Farrin Marouf Sofian, Felix Draxler, Theofanis Karaletsos, and Stephan Mandt. Variational Control for Guidance in Diffusion Models. International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research.
Questions for authors
See Weakness
10 comments
W2 (Part 1/2)
W2 Rejection sampling at equal compute (Part 1/2)
What the reviewers mention is essentially rejection sampling. The reviewer is right that, with enough candidates, selecting by the oracle can match any target whose classes all have nonzero probability under the base model. We add this baseline and clarify our method's applicability and strengths in the next part.
2.1 Rejection sampling
- Setup. To output images with the target class proportions, we draw ordinary (unguided) samples from the base model, label each with the same classifier, and keep a sample only while its class is below its target count; the rest are discarded. At equal compute budget with the settings reported in the paper, rejection sampling may draw 30x more samples than ours (see Part 2 for compute cost). Beyond the prescribed budget, classes still short are filled with previous discarded samples, so the output always has samples in total. With the acceleration above, one our method image costs only about 16 ordinary samples; we keep the budget at 31, which matches the unaccelerated our method runs being compared and is the more generous choice for rejection sampling.
- Results. At equal compute, rejection sampling matches the target proportions as well as or better than our method on all 11 FFHQ targets: TV on 10 of them, on the 24-class custom joint. However, our method's sample quality appears to be better: FID is lower on 11/11 targets by –; CMMD is lower on 10/11 tested settings, We attribute this in part to our method optimizing each sampling trajectory rather than only filtering ordinary samples.
FFHQ: rejection sampling at matched compute vs Ours
| Target | Attribute | Method | TV | FID | CMMD |
|---|---|---|---|---|---|
| Uniform | age | Rejection sampling | .0000 | 48.88 | .724 |
| Ours | .0230 | 45.18 | .656 | ||
| Uniform | race | Rejection sampling | .0000 | 51.30 | .811 |
| Ours | .0280 | 47.59 | .836 | ||
| Custom 1 | age | Rejection sampling | .0000 | 51.28 | .907 |
| Ours | .0490 | 46.14 | .791 | ||
| Custom 2 | race | Rejection sampling | .0007 | 51.62 | .882 |
| Ours | .0530 | 46.53 | .855 | ||
| Uniform | age gender race | Rejection sampling | .0000 | 47.88 | .826 |
| Ours | .1120 | 41.28 | .762 | ||
| CustomJoint | age gender race | Rejection sampling | .0216 | 47.42 | .801 |
| Ours | .1420 | 42.19 | .747 |
W2 (Part 2/2)
W2 Rejection sampling (Part 2/2)
W2.2 When our method is conceptually the better tool
"Sufficiently many" is the catch. Our method is the better tool when the target asks for much more of a class that the base model rarely produces. We model that as follows:
- Let be the probability of attribute class among the base model’s generated distribution, the target, and the number of samples rejection sampling may draw per final sample (i.e., the compute budget multiplier). Among samples about are in class , but are needed, so every class with ends up short, and the TV to the target is . This is zero only if for every class. The desired budget is therefore set by the class that the target asks for most relative to how often the model produces it (e.g. 25% requested, 1% produced: ), and these ratios multiply when several independent attributes are constrained jointly. In contrast, our method's cost does not depend on , since it steers samples toward the needed classes instead of waiting for them to appear.
- We add a 2D toy example for illustration in Appendix G, pp. 68 in the revised paper. We set the base model's distribution as a zero-mean Gaussian. We set two attribute classes: a small disc away from the mode, which the base model samples with probability , and any samples outside the disc belong to the other class. We demonstrate with a uniform target . Both methods get the same compute, so rejection sampling draws (see below) samples per output. As the disc moves outward, varies from to . Rejection sampling works better while , and our method becomes better from that point, as the law we analyzed in the above point. For , our method's TV is – against – for rejection sampling. At , for example, rejection sampling places 1.8% of its samples in the disc and our method 49.1%. The visualizations and the full table are in the revised paper Appendix G.
2.3 Compute cost
- Compute cost. Our method's compute cost in the reported setups was about 15x of the guidance baselines and about 31x of vanilla generation using the reported configurations. However, we further apply the following to make our method 1.94x faster (tested on CIFAR-100), with both alignment and sample quality roughly the same:
- Caching. Each E-MSA iteration first rolls the sampler forward and then runs the adjoint (backward) sweep, which needs the denoiser's vector–Jacobian product at every sampling step. The original implementation discards the denoiser's intermediate activations during the forward rollout and recomputes them in the backward sweep. We instead keep them, i.e. the per-step computation graph of the denoiser, so the backward sweep computes each vector–Jacobian product directly. This removes one of the three network passes per step and iteration.
- Fewer iterations. The fixed points of the E-MSA update depend on and only through (Alg. 1): the update is a projected gradient step of size on the same -regularised objective (Proposition 3.2). Prop. 3.2 bounds the smallest gradient-mapping norm after iterations by , so doubling the step at the same lets us stop after 7 instead of 10 iterations, with TV and FID roughly the same. However, the step cannot grow without limit: the same result requires , where is the gradient-Lipschitz constant of the discretised objective (Assumption C.6, Lemma C.10). depends on the Lipschitz constants of the denoiser, oracle and terminal cost and is not available in closed form. Equivalently, a larger shrinks the proximal weight of the E-MSA subproblem (Eq. (11)), which keeps each update close to the previous trajectory. We therefore treat as a tunable hyperparameter and report only the doubled step size that we verified.
- Together, these bring the compute budget multiplier from 31x to around 16x of the vanilla generation.
W1
We thank the reviewer for their detailed reviews. We respond to each point below.
W1 Why unconditional model?
Before addressing the individual concerns, we distinguish three regimes of controlled diffusion generation.
Our method is intended primarily for Regime 3 and is potentially applicable to Regime 2. It is not intended to replace direct, well-calibrated conditional sampling in Regime 1. The reviewer's CIFAR-100 example belongs to Regime 1: a class-conditional model sampled with class quotas is the right tool there.
References
[1] Song, J. et al. (2023). Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation. ICML 2023.
[2] Ye, H. et al. (2024). TFG: Unified Training-Free Guidance for Diffusion Models. NeurIPS 2024.
[3] Bansal, A. et al. (2024). Universal Guidance for Diffusion Models. ICLR 2024.