acceptodds
Under review as a conference paper at ICLR 2027

Inference-Time Attribute Distribution Alignment for Unconditional Diffusion

Abstract

Inference-time controllable generation is essential for real-world applications of unconditional diffusion models. However, most existing techniques focus on individual samples, struggling in applications that require the sample population to follow specific attribute distributions (e.g., demographic balance or semantic proportions). We formalize this setting as the inference-time attribute distributional alignment problem for pretrained unconditional diffusion models. To address this, we cast inference-time attribute distributional alignment as an optimal control problem over the reverse diffusion process, viewing the process as the rollout of a dynamical system and augmenting it with additive, time-dependent perturbations as control. We solve for the perturbations using an optimal-control-based algorithm to optimize a differentiable distribution-matching objective while penalizing control effort to preserve data fidelity. Experiment results in image generation demonstrate that our proposed plug-and-play approach can better align attribute distributions to diverse and flexible test-time targets compared to baselines, without retraining or finetuning the pretrained diffusion model.

Primary area:
generative models
Keywords:
diffusion, distribution alignment, test-time compute, optimal control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dRf1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper focus on the attribute distribution alignment problem of pretrained unconditional diffusion models in the inference stage, aiming to make the attribute distribution of generated samples matches a target distribution without fine-tuning the generative model. The authors model the reverse diffusion process as an optimal control problem, taking the KL divergence of the batch sample attribute distribution as the cost, and introduce a control regularization term, iteratively optimizing the control variables via Pontryagin's maximum principle and the E-MSA algorithm. In addition, the authors provide theoretical analysis on the algorithm's convergence and attribute distribution alignment. Experiments cover semantic classes control on CIFAR-100 and human-face attributes control on FFHQ. Experimental results show that this method can more effectively match the target attribute distribution compared to the selected baselines, while maintaining good generation quality.

Critical strengths

1.The problem modeling aligns well with the distribution matching. Since attribute distribution matching is essentially a group-level objective, the authors did not simply decompose it into independent per-sample rewards, but instead used batch-level KL divergence as the terminal cost to explicitly couple multiple generated samples, and introduced control regularization to balance distribution alignment and generation quality. This modeling approach is natural and reasonable for this problem.

Critical weaknesses

1.The paper does not sufficiently justify why attribute distribution matching should be addressed by controlling pretrained unconditional diffusion models. For semantic classes distribution matching on CIFAR-100, a more direct and natural solution would be using a class-conditional diffusion model and sample according to the desired class distribution. Similarly, for human face generation, text-to-image models with carefully designed text prompts could be used to control the proportions of different attributes.

2.When restricted to pretrained unconditional diffusion models, a straightforward strategy could be considered: generate a sufficiently large pool of candidate images and select samples using an attribute classifier to match the target distribution. Theoretically, as long as every target class has nonzero probability under the original generative distribution, sufficientl independent samples can provide the required number of images of each class, allowing arbitrary target classes distributions to be matched.

3.The experimental comparisons do not adequately cover closely related methods or simpler alternatives. The authors discuss OC-Flow [1] and DTM [2] in the Related Work section, both of which employ optimal control to steer the sampling trajectories of pretrained generative models. However, the paper does not sufficiently clarify the differences from these existing optimal control frameworks, nor does it provide comparisons under a unified batch-level distribution matching objective. Consequently, it remains unclear whether the contribution primarily amounts to replacing the terminal cost function in an existing optimal control framework with a batch-level distribution matching cost.The experiments lack simple but important baselines, such as generate-then-select mentioned in 2, as well as reference comparisons with conditional generation combined with class-quota sampling. According to the complexity analysis in Appendix B.3, the proposed method requires multiple forward simulations and backward gradient computations. With approximately 10 optimization iterations, its generative-model computation is roughly comparable to that of several dozen standard unconditional sampling runs. This raises a natural question: under the same computational budget, could one simply generate more candidate images and select them according to the desired attribute proportions using the Attribute Oracle, achieving comparable or even better distribution alignment?

4.The paper uses the same Attribute Oracle for generation optimization and attribute distribution evaluation, potentially introducing oracle-specific evaluation bias. Although the appendix includes ablation studies using ResNet and ViT as alternative oracles, it does not provide independent cross-oracle evaluations, such as optimizing generation with ResNet and evaluating the resulting attribute distribution with ViT. The current experiments cannot exclude the possibility that the method primarily exploits the decision boundaries of a particular classifier, rather than achieving reliable alignment of the underlying semantic attribute distribution.

References

[1] Luran Wang, Chaoran Cheng, Yizhen Liao, Yanru Qu, and Ge Liu. Training Free Guided Flow-Matching with Optimal Control. International Conference on Learning Representations (ICLR), 2025.

[2] Kushagra Pandey, Farrin Marouf Sofian, Felix Draxler, Theofanis Karaletsos, and Stephan Mandt. Variational Control for Guidance in Diffusion Models. International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research.

Questions for authors

See Weakness

10 comments

We thank the reviewer for their detailed reviews. We respond to each point below.

W1 Why unconditional model?

Before addressing the individual concerns, we distinguish three regimes of controlled diffusion generation.

  1. A native, calibrated conditioning interface exists for the attribute of interest. For example, a class-conditional diffusion model exposes exactly the desired classes, and sampling class labels in the specified proportions is likely to produce the same proportions under the oracle.
  2. The generator is conditional, but the attribute of interest is external or orthogonal to its conditioning interface. For example, a model may accept text prompts while the target attribute is defined by a separate safety-related, behavioural or scientific evaluator, and cannot be reliably specified through prompts alone. Our formulation may conceptually apply in this regime under suitable assumptions, but we make no empirical claims about it.
  3. The pretrained generator has no conditioning interface for the attribute. This is the regime studied and evaluated in the paper.

Our method is intended primarily for Regime 3 and is potentially applicable to Regime 2. It is not intended to replace direct, well-calibrated conditional sampling in Regime 1. The reviewer's CIFAR-100 example belongs to Regime 1: a class-conditional model sampled with class quotas is the right tool there.

  • The abstraction. Our claim is not that unconditional diffusion is generally preferable for image generation. The relevant abstraction is that the inference-time attribute is not built into the generator's conditioning interface, as in Regimes 2 and 3. This setup is well established in training-free guidance [1-3], where a frozen generator is controlled by an independently supplied predictor of a property or objective, without retraining the generator. One motivation is that practitioners can change the attribute of interest at inference time while keeping the generator fixed. Our work differs from most existing guidance methods in controlling the population-level attribute distribution without a set of reference samples but only an oracle model, rather than having each sample satisfy a condition independently.
  • Image experiments only as testbeds. We clarify that our method conceptually does not limit to image generation and that we only use image experiments as controlled, interpretable testbeds for the formulation. We chose them for accessibility and reproducibility (public datasets and pretrained weights are widely available, whereas acquiring specialized domain data and models is beyond the scope of this work) and for interpretability (the tasks and visual results are easy to perceive; CIFAR-100 lets us vary the support size of the target distribution conveniently via meta5 / coarse / fine, and FFHQ allows interpretable single- and joint-attribute evaluations).

References
[1] Song, J. et al. (2023). Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation. ICML 2023.
[2] Ye, H. et al. (2024). TFG: Unified Training-Free Guidance for Diffusion Models. NeurIPS 2024.
[3] Bansal, A. et al. (2024). Universal Guidance for Diffusion Models. ICLR 2024.

AuthorsOpenReview ↗

W2 (Part 1/2)

W2 Rejection sampling at equal compute (Part 1/2)

What the reviewers mention is essentially rejection sampling. The reviewer is right that, with enough candidates, selecting by the oracle can match any target whose classes all have nonzero probability under the base model. We add this baseline and clarify our method's applicability and strengths in the next part.

2.1 Rejection sampling

  • Setup. To output NN images with the target class proportions, we draw ordinary (unguided) samples from the base model, label each with the same classifier, and keep a sample only while its class is below its target count; the rest are discarded. At equal compute budget with the settings reported in the paper, rejection sampling may draw 30x more samples than ours (see Part 2 for compute cost). Beyond the prescribed budget, classes still short are filled with previous discarded samples, so the output always has NN samples in total. With the acceleration above, one our method image costs only about 16 ordinary samples; we keep the budget at 31, which matches the unaccelerated our method runs being compared and is the more generous choice for rejection sampling.
  • Results. At equal compute, rejection sampling matches the target proportions as well as or better than our method on all 11 FFHQ targets: TV ≤7×10−4\le 7 \times 10^{-4} on 10 of them, 0.0220.022 on the 24-class custom joint. However, our method's sample quality appears to be better: FID is lower on 11/11 targets by 2.92.9–6.66.6; CMMD is lower on 10/11 tested settings, We attribute this in part to our method optimizing each sampling trajectory rather than only filtering ordinary samples.

FFHQ: rejection sampling at matched compute vs Ours

TargetAttributeMethodTV ↓\downarrowFID ↓\downarrowCMMD ↓\downarrow
UniformageRejection sampling.000048.88.724
Ours.023045.18.656
UniformraceRejection sampling.000051.30.811
Ours.028047.59.836
Custom 1ageRejection sampling.000051.28.907
Ours.049046.14.791
Custom 2raceRejection sampling.000751.62.882
Ours.053046.53.855
Uniformage ×\times gender ×\times raceRejection sampling.000047.88.826
Ours.112041.28.762
CustomJointage ×\times gender ×\times raceRejection sampling.021647.42.801
Ours.142042.19.747
AuthorsOpenReview ↗

W2 (Part 2/2)

W2 Rejection sampling (Part 2/2)

W2.2 When our method is conceptually the better tool

"Sufficiently many" is the catch. Our method is the better tool when the target asks for much more of a class that the base model rarely produces. We model that as follows:

  • Let pcp_c be the probability of attribute class cc among the base model’s generated distribution, qcq_c the target, and rr the number of samples rejection sampling may draw per final sample (i.e., the compute budget multiplier). Among rNrN samples about r pcNr\,p_c N are in class cc, but qcNq_c N are needed, so every class with r pc<qcr\,p_c < q_c ends up short, and the TV to the target is TV≈∑cmax⁡{qc−r pc, 0}\mathrm{TV} \approx \sum_c \max\{q_c - r\,p_c,\ 0\}. This is zero only if r≥qc/pcr \ge q_c/p_c for every class. The desired budget is therefore set by the class that the target asks for most relative to how often the model produces it (e.g. 25% requested, 1% produced: r∗≥25r^\ast \ge 25), and these ratios multiply when several independent attributes are constrained jointly. In contrast, our method's cost does not depend on pcp_c, since it steers samples toward the needed classes instead of waiting for them to appear.
  • We add a 2D toy example for illustration in Appendix G, pp. 68 in the revised paper. We set the base model's distribution as a zero-mean Gaussian. We set two attribute classes: a small disc away from the mode, which the base model samples with probability pp, and any samples outside the disc belong to the other class. We demonstrate with a uniform target [1/2,1/2][1/2, 1/2]. Both methods get the same compute, so rejection sampling draws r=16r = 16 (see below) samples per output. As the disc moves outward, pp varies from 10−110^{-1} to 10−510^{-5}. Rejection sampling works better while p≳q/r=1/32p \gtrsim q/r = 1/32, and our method becomes better from that point, as the law we analyzed in the above point. For 10−5≤p≤0.0210^{-5} \le p \le 0.02, our method's TV is 0.0060.006–0.0570.057 against 0.180.18–0.500.50 for rejection sampling. At p=10−3p = 10^{-3}, for example, rejection sampling places 1.8% of its samples in the disc and our method 49.1%. The visualizations and the full table are in the revised paper Appendix G.

2.3 Compute cost

  • Compute cost. Our method's compute cost in the reported setups was about 15x of the guidance baselines and about 31x of vanilla generation using the reported configurations. However, we further apply the following to make our method 1.94x faster (tested on CIFAR-100), with both alignment and sample quality roughly the same:
    • Caching. Each E-MSA iteration first rolls the sampler forward and then runs the adjoint (backward) sweep, which needs the denoiser's vector–Jacobian product at every sampling step. The original implementation discards the denoiser's intermediate activations during the forward rollout and recomputes them in the backward sweep. We instead keep them, i.e. the per-step computation graph of the denoiser, so the backward sweep computes each vector–Jacobian product directly. This removes one of the three network passes per step and iteration.
    • Fewer iterations. The fixed points of the E-MSA update depend on η\eta and ξ\xi only through ρ=(1−ξ)/η\rho = (1-\xi)/\eta (Alg. 1): the update is a projected gradient step of size η\eta on the same ρ\rho-regularised objective (Proposition 3.2). Prop. 3.2 bounds the smallest gradient-mapping norm after II iterations by 2 (Jh(U(0))−Jhinf⁡)/(ηI)2\,(J_h(U^{(0)}) - J_h^{\inf})/(\eta I), so doubling the step at the same ρ\rho lets us stop after 7 instead of 10 iterations, with TV and FID roughly the same. However, the step cannot grow without limit: the same result requires η≤1/L\eta \le 1/L, where LL is the gradient-Lipschitz constant of the discretised objective (Assumption C.6, Lemma C.10). LL depends on the Lipschitz constants of the denoiser, oracle and terminal cost and is not available in closed form. Equivalently, a larger η\eta shrinks the proximal weight γ=1/η−ρ\gamma = 1/\eta - \rho of the E-MSA subproblem (Eq. (11)), which keeps each update close to the previous trajectory. We therefore treat η\eta as a tunable hyperparameter and report only the doubled step size that we verified.
    • Together, these bring the compute budget multiplier from 31x to around 16x of the vanilla generation.
Reviewer_86P4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies inference-time attribute distribution alignment for pre-trained unconditional diffusion models. It proposes an optimal-control framework that adjusts diffusion trajectories to match a target attribute distribution across generated samples, without retraining the model. Experiments on CIFAR-100 and FFHQ show improvements in distribution alignment over existing guidance-based baselines, while generally preserving image quality.

Critical strengths

The paper addresses a meaningful fairness problem through inference time alignment. The optimal-control formulation is principled, and the consistent improvements across different diffusion architectures, attribute distributions, and experimental settings provide strong empirical support.

Critical weaknesses

The practical advantage over simpler distribution-matching approaches remains unclear. Since the objective is to match population-level attribute proportions, a natural baseline is to generate a larger pool of unconditional samples (e.g. from EDM) and select those matching the desired distribution with high quality. Given the substantial computational overhead of the proposed iterative optimization, it is difficult to determine whether the additional complexity is justified.

Questions for authors
  1. How does the method compare with unconditional sampling from EDM followed by attribute-aware selection? Can it achieve comparable distribution alignment and image quality at a lower cost?

  2. How reliable is the measured attribute alignment? Could the authors evaluate generated samples using an independent attribute classifier (different from the optimization oracle), and assess image quality separately from distribution matching?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 86P4

We thank the reviewer for their time and efforts spent on the reviews and for their positive assessment.

W1/Q1 Rejection sampling at equal compute cost (Part 1/2)

The generate-then-select baseline is essentially rejection sampling. We add comparisons to this baseline and clarify our method’s applicability and strengths.

  • Setup. To output NN images with the target class proportions, we draw ordinary (unguided) samples from the base model, label each with the same classifier, and keep a sample only while its class is below its target count; the rest are discarded. At equal compute budget with the settings reported in the paper, rejection sampling may draw 30x more samples than ours (see Part 2 for compute cost). Beyond the prescribed budget, classes still short are filled with previous discarded samples, so the output always has NN samples in total. With the acceleration discussed below, one our method image costs only about 16 ordinary samples; we keep the budget at 31, which matches the unaccelerated our method runs being compared and is the more generous choice for rejection sampling.
  • Results. At equal compute, rejection sampling matches the target proportions as well as or better than our method on all 11 FFHQ targets: TV ≤7×10−4\le 7 \times 10^{-4} on 10 of them, 0.0220.022 on the 24-class custom joint. However, our method's sample quality appears to be better: FID is lower on 11/11 targets by 2.92.9–6.66.6; CMMD is lower on 10/11 tested settings. We attribute this in part to our method optimizing each sampling trajectory rather than only filtering ordinary samples.

FFHQ: rejection sampling at matched compute vs Ours

TargetAttributeMethodTV ↓\downarrowFID ↓\downarrowCMMD ↓\downarrow
UniformageRejection sampling.000048.88.724
Ours.023045.18.656
UniformraceRejection sampling.000051.30.811
Ours.028047.59.836
Custom 1ageRejection sampling.000051.28.907
Ours.049046.14.791
Custom 2raceRejection sampling.000751.62.882
Ours.053046.53.855
Uniformage ×\times gender ×\times raceRejection sampling.000047.88.826
Ours.112041.28.762
CustomJointage ×\times gender ×\times raceRejection sampling.021647.42.801
Ours.142042.19.747

W1/Q1 Rejection sampling at equal compute cost (Part 2/2)

W1/Q1.2 When our method is conceptually the better tool

Our method conceptually works better when the target asks for much more of an attribute that the base model rarely produces. We model that as follows:

  • Let pcp_c be the probability of attribute class cc among the base model’s generated distribution, qcq_c the target, and rr the number of samples rejection sampling may draw per final sample (i.e., the compute budget multiplier). Among rNrN samples about r pcNr\,p_c N are in class cc, but qcNq_c N are needed, so every class with r pc<qcr\,p_c < q_c ends up short, and the TV to the target is TV≈∑cmax⁡{qc−r pc, 0}\mathrm{TV} \approx \sum_c \max\{q_c - r\,p_c,\ 0\}. This is zero only if r≥qc/pcr \ge q_c/p_c for every class. The desired budget is therefore set by the class that the target asks for most relative to how often the model produces it (e.g. 25% requested, 1% produced: r∗≥25r^\ast \ge 25), and these ratios multiply when several independent attributes are constrained jointly. In contrast, our method's cost does not depend on pcp_c, since it steers samples toward the needed classes instead of waiting for them to appear.
  • We add a 2D toy example for illustration in Appendix G, pp. 68 in the revised paper. We set the base model's distribution as a zero-mean Gaussian, so every class probability is exact. We set two attribute classes: a small disc away from the mode, which the base model samples with probability pp, and any samples outside the disc belong to the other class. We demonstrate with a uniform target [1/2,1/2][1/2, 1/2]. Both methods get the same compute, so rejection sampling draws r=16r = 16 (see below) samples per output. As the disc moves outward, pp varies from 10−110^{-1} to 10−510^{-5}. Rejection sampling works better while p≳q/r=1/32p \gtrsim q/r = 1/32, and our method becomes better from that point, as the law we analyzed in the above point. For 10−5≤p≤0.0210^{-5} \le p \le 0.02, our method's TV is 0.0060.006–0.0570.057 against 0.180.18–0.500.50 for rejection sampling. At p=10−3p = 10^{-3}, for example, rejection sampling places 1.8% of its samples in the disc and our method 49.1%. The visualizations and the full table are in the revised paper Appendix G.

1.3 Compute cost

  • Compute cost. Our method's compute cost in the reported setups was about 15x of the guidance baselines and about 31x of vanilla generation using the reported configurations. However, we further apply the following to make our method 1.94x faster (tested on CIFAR-100), with both alignment and sample quality roughly the same:
    • Caching. Each E-MSA iteration first rolls the sampler forward and then runs the adjoint (backward) sweep, which needs the denoiser's vector–Jacobian product at every sampling step. The original implementation discards the denoiser's intermediate activations during the forward rollout and recomputes them in the backward sweep. We instead keep them, i.e. the per-step computation graph of the denoiser, so the backward sweep computes each vector–Jacobian product directly. This removes one of the three network passes per step and iteration.
    • Fewer iterations. The fixed points of the E-MSA update depend on η\eta and ξ\xi only through ρ=(1−ξ)/η\rho = (1-\xi)/\eta (Alg. 1): the update is a projected gradient step of size η\eta on the same ρ\rho-regularised objective (Proposition 3.2). Prop. 3.2 bounds the smallest gradient-mapping norm after II iterations by 2 (Jh(U(0))−Jhinf⁡)/(ηI)2\,(J_h(U^{(0)}) - J_h^{\inf})/(\eta I), so doubling the step at the same ρ\rho lets us stop after 7 instead of 10 iterations, with TV and FID roughly the same. However, the step cannot grow without limit: the same result requires η≤1/L\eta \le 1/L, where LL is the gradient-Lipschitz constant of the discretised objective (Assumption C.6, Lemma C.10). LL depends on the Lipschitz constants of the denoiser, oracle and terminal cost and is not available in closed form. Equivalently, a larger η\eta shrinks the proximal weight γ=1/η−ρ\gamma = 1/\eta - \rho of the E-MSA subproblem (Eq. (11)), which keeps each update close to the previous trajectory. We therefore treat η\eta as a tunable hyperparameter and report only the doubled step size that we verified.
    • Together, these bring the compute budget multiplier from 31x to around 16x of the vanilla generation.
AuthorsOpenReview ↗

Q2 (Part 1/2)

Q2 How reliable is the measured alignment? (Part 1/2)

2.1 Independent classifier evaluation

  • The reported CIFAR results in the initial submission were already evaluated under a held-out ResNet classifier different from the one used from steering.
  • Setup. We re-evaluated every FFHQ sample with held-out RetNet and ViT classifiers that are not used for steering.
  • Results: The following tables report re-evaluated TV metrics; FID and CMMD do not depend on the classifier and are repeated for reference. Overall, our method’s advantage in alignment persists in most tested settings. In two cells on FFHQ, two guidance baselines achieves slightly better alignment ( <.01<.01 TV), but at the cost of substantially worse sample quality as indicated by much higher FID and CMMD scores.

FFHQ

TargetAttributeMethodTV, held-out ResNet ↓\downarrowTV, ViT ↓\downarrowFID ↓\downarrowCMMD ↓\downarrow
UniformageDDIM.264.34047.18.750
PG-DPS.199.407114.41.38
Smpl+DPS.121.368130.71.488
LFM.333.35145.34.751
Ours.129.25245.18.656
Ours-F.202.26143.60.643
UniformgenderDDIM.0281.032347.28.705
PG-DPS.0604.015675.67.899
Smpl+DPS.0687.11854.97.819
LFM.151.15344.07.698
Ours.00417.017744.97.586
Ours-F.0615.07542.67.637
UniformraceDDIM.548.61052.38.862
PG-DPS.457.622101.81.264
Smpl+DPS.393.58969.561.071
LFM.568.61648.41.835
Ours.221.46847.59.836
Ours-F.392.53346.12.681
Custom 1ageDDIM.472.54849.65.959
PG-DPS.447.644123.11.620
Smpl+DPS.274.45095.051.285
LFM.542.55951.95.972
Ours.185.31746.14.791
Ours-F.291.38548.25.793
Custom 1genderDDIM.272.26848.58.979
PG-DPS.366.322129.61.759
Smpl+DPS.122.087570.641.101
LFM.149.14745.87.915
Ours.0813.12946.52.804
Ours-F.0438.061545.76.818
Custom 1raceDDIM.398.46048.71.861
PG-DPS.431.526165.31.917
Smpl+DPS.317.47361.741.050
LFM.418.46648.80.861
Ours.221.42543.03.741
Ours-F.267.40244.73.683
Custom 2ageDDIM.277.34951.39.990
PG-DPS.263.37290.791.338
Smpl+DPS.152.382114.61.545
LFM.335.36251.50.990
Ours.109.21746.25.782
Ours-F.201.27747.83.814
Custom 2genderDDIM.228.23255.74.969
PG-DPS.0313.0802156.91.890
Smpl+DPS.152.20160.621.051
LFM.351.35356.161.011
Ours.025.07548.69.803
Ours-F.170.21051.22.884
Custom 2raceDDIM.687.74951.45.949
PG-DPS.457.858253.23.113
Smpl+DPS.464.71072.941.163
LFM.707.75551.12.959
Ours.309.56746.53.855
Ours-F.473.65846.09.719
Uniformage×gender×raceDDIM.564.63846.26.912
PG-DPS.518.664127.01.684
Smpl+DPS.551.747198.22.591
LFM.590.64446.20.906
Ours.266.51641.28.762
Ours-F.447.58141.54.703
CustomJointage×gender×raceDDIM.510.57344.28.895
PG-DPS.566.58874.771.192
Smpl+DPS.443.54989.511.252
LFM.526.55744.05.858
Ours.283.41842.19.747
Ours-F.299.44638.65.564
Reviewer_gbWgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of aligning the attribute distribution of samples generated by pretrained unconditional diffusion models with a target distribution specified at test time (e.g., demographic balance). The authors formulate this problem as an optimal control problem over the reverse diffusion process, where a time-dependent control is optimized to minimize the KL divergence between the attribute distribution of a batch and the target, with a control-effort penalty to preserve sample quality. The method is training-free and comes with convergence and alignment guarantees. Experiments on CIFAR-100 and FFHQ show better alignment than the baselines while keeping FID close to the base model.

Critical strengths
  • The problem formulation is interesting and well-motivated, as distribution-level objectives such as fairness cannot be expressed by a per-sample reward.

  • The optimal control formulation gives an explicit trade-off between alignment and sample quality, and changing the target only changes the terminal cost without any retraining.

  • The paper is easy to follow and generally well written, and the theoretical results support the proposed algorithm.

  • The method improves alignment considerably over the included baselines while preserving FID, also for joint attributes and latent flow matching.

Critical weaknesses

My main concern regarding this work is its originality and the experimental comparison. The solver largely follows the E-MSA approach used in OC-Flow (Wang et al., 2025) applied to a batch of samples, so the main new component is the batch-level terminal cost. Moreover, the claim that few generic methods can align pretrained diffusion models to test-time target distributions in a plug-and-play fashion should be revisited, since training-free distribution-level guidance has already been explored in prior work, e.g., MMD Guidance [1]. Below I outline additional concerns:

  • The paper misses relevant baselines and related works. MMD Guidance [1] is a closely related baseline, as it is training-free, applies to unconditional diffusion models, and aligns a batch of generated samples with a target distribution. It can be applied almost directly here, since the paper already samples reference sets according to the target distribution for computing FID (footnote 5). In addition, for the Uniform (fairness) targets, diversity guidance methods such as c-VSG [4], SPARKE [2], and SPELL [3] are natural baselines, in the same way PG is adapted in the paper. A simple resampling baseline (generating i.i.d. samples, labeling them with the oracle, and subsampling to match the target) would also be important for understanding the benefit of the proposed method.

  • The comparison with PG does not seem fair. The guidance weight of PG is fixed to w = 4.0 for all CIFAR targets (Table 7) without a time-dependent schedule, while the proposed method is tuned per target (Table 6). The very high FID of PG on faces (e.g., 253.2 with w = 100) suggests that the guidance is too strong. Also, the proposed method uses 10 forward/backward passes through the sampler while PG uses a single pass, so a comparison under matched compute is needed.

  • It is unclear whether the same classifier is used for steering and evaluation. If so, the improvements may partly come from exploiting the classifier rather than changing the attribute itself. For example, in Table 3 (gender, Custom-2) the 7:3 ratio is matched exactly (TV = 1.2e-8). Evaluating with an independent classifier (or CLIP zero-shot) would make the results more convincing.

  • The diversity of samples within each attribute group is not evaluated. Pushing samples toward under-represented attributes may reduce diversity within groups, which FID does not clearly capture. Using diversity metrics such as RKE [5] and Vendi [6] for each attribute group could help better show that alignment does not come at the cost of diversity.

  • The method requires generating a large batch jointly (around 5-10 times the support size), which may limit its applicability in practical settings where samples are generated one at a time. In addition, the experiments are limited to CIFAR-100 and FFHQ-256, and it is unclear whether the method scales to larger models such as Stable Diffusion.

Minor weaknesses:

  • The JS values in Tables 2 and 3 seem inconsistent: for LFM and OURS-F they match the JS divergence, while for the other rows they match the JS distance (e.g., Table 3, Uniform-age: OURS has χ² = 6.4e-4 and JS = 0.018, while OURS-F has χ² = 3.6e-4 and JS = 1.8e-4). This makes OURS-F look much better than it is.
  • Providing a memory and runtime comparison between the proposed method and the baselines would better clarify the computational cost of the method.
  • Typos: "Guassian" (Appendix E.1) and a missing decimal point in Table 3 (FD = "37").

[1] Mahdizadeh Sani et al. "MMD Guidance: Training-Free Distribution Adaptation for Diffusion Models via Maximum Mean Discrepancy Guidance", ICML 2026

[2] Jalali et al. "SPARKE: Scalable Prompt-Aware Diversity and Novelty Guidance in Diffusion Models via RKE Score", NeurIPS 2025

[3] Kirchhof et al. "Sparse Repellency for Shielded Generation in Text-to-Image Diffusion Models", ICML 2025

[4] Askari Hemmat et al. "Improving Geo-diversity of Generated Images with Contextualized Vendi Score Guidance", ECCV 2024

[5] Jalali et al. "An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions", NeurIPS 2023

[6] Friedman and Dieng. "The Vendi Score: A Diversity Evaluation Metric for Machine Learning", TMLR 2023

Questions for authors
  • How was the guidance weight of PG chosen? Did you try a time-dependent schedule for it?

  • Are the reported metrics computed on hard labels or on averaged softmax probabilities?

  • How does the method compare with MMD Guidance, and what happens if the KL terminal cost is replaced by an MMD-based cost?

  • In Appendix E.1, the alignment degrades when the batch size grows beyond a certain point. Why does this happen, since a larger batch should give a better estimate of the attribute distribution?

  • Is it possible to provide a recipe for sequential sample generation rather than parallel generation? This could have broader practical applications.

11 comments
AuthorsOpenReview ↗

Response to Reviewer gbWg

We first thank the reviewer for their time and for the detailed review. We answer the questions first, each together with the weakness it may associates, and then the remaining weaknesses and minor points.

AuthorsOpenReview ↗

Q1/W2 PG Tuning (Part 1/2)

Q1/W2 PG Tuning (Part 1/2)

1.1 Fixed PG guidance strength

On both benchmarks we performed grid search on the guidance strength. On FFHQ, the guidance strength was chosen as the one that produced best TV in the grid, as we disclosed in the original submission (line 2653); this partially explains the high FID. On CIFAR, we acknowledge that the same rule was not applied, and we update the results below under the same guidance strength selection rule. The following table shows that despite further choosing guidance weights for PG, our method still delivers better alignment and preserves better sample quality.

TargetLevelMethodwwTV ↓\downarrowFID ↓\downarrow
Uniformmeta5PG-DPS1.047416.22
Ours—.028113.07
ZigZagmeta5PG-DPS1.096216.29
Ours—.054413.97
Gaussianmeta5PG-DPS5.148123.08
Ours—.136116.48
UniformcoarsePG-DPS1.079413.08
Ours—.069012.62
ZigZagcoarsePG-DPS4.133015.74
Ours—.102914.84
GaussiancoarsePG-DPS9.187619.56
Ours—.176415.72
UniformfinePG-DPS1.207916.43
Ours—.140512.59
ZigZagfinePG-DPS1.255817.55
Ours—.171312.60
GaussianfinePG-DPS1.255117.97
Ours—.184413.74
AuthorsOpenReview ↗

Q1/W2 PG Tuning (Part 2/2)

Q1/W2 PG Tuning (Part 2/2)

1.2 Time-dependent guidance schedule

We acknowledge that a time-dependent guidance strength schedule was not attempted. We now add a log-interpolation guidance strength schedule, following Appendix D.2 of the PG paper.

  • Setups.
    • CIFAR-100: we try every combination of two different start and end values from {1,4,16}\{1, 4, 16\} (six schedules in total) and kept the one with the lowest mean TV over the six meta5 and coarse targets. The selected schedule raises the weight from 1 to 16, i.e. weak guidance early and strong guidance late. Then, we run it on all nine targets with the same setups as in Table 1.
    • FFHQ: The high FID of fixed-ww PG on faces confirms the reviewer's point that the guidance was too strong there. We keep the same shape (the final weight is 16 times the initial one) and tune only its overall strength, with geometric-mean weight wˉ∈{0.5,1,2,4,8}\bar w \in \{0.5, 1, 2, 4, 8\}, which is much lower than the fixed weights. The same sampler and other settings are used as in Tables 2–3.
  • Results. The following tables contain the time-dependent guidance strengths results from PG, compared to ours.
    • CIFAR-100: The schedule helps PG (better than fixed ww on TV, JS, χ2\chi^2 and FD in 8/9 cells, FID in 6/9). Despite the improvement, our method still performs better on the tested settings.
    • FFHQ: Our method has better FID on all 11 targets and better TV, JS, χ2\chi^2 and FD on 10 out of 11 tested targets. The only exception is Uniform-age, where scheduled PG at wˉ=8\bar w = 8 aligns slightly better (TV 0.0180.018 vs .023.023) while at 6.8 worse FID.

CIFAR-100: PG-DPS w/ time-dependent schedule vs Ours

TargetLevelMethodTV ↓\downarrowFID ↓\downarrow
Uniformmeta5PG-DPS, scheduled 1→161 \to 16.050718.8
PG-DPS, w=4w=4.069222.4
Ours.028113.1
ZigZagmeta5PG-DPS, scheduled 1→161 \to 16.10418.4
PG-DPS, w=4w=4.11322.0
Ours.054414.0
Gaussianmeta5PG-DPS, scheduled 1→161 \to 16.14919.1
PG-DPS, w=4w=4.15421.3
Ours.13616.5
UniformcoarsePG-DPS, scheduled 1→161 \to 16.085215.5
PG-DPS, w=4w=4.080515.2
Ours.069012.6
ZigZagcoarsePG-DPS, scheduled 1→161 \to 16.12715.9
PG-DPS, w=4w=4.13315.7
Ours.10314.8
GaussiancoarsePG-DPS, scheduled 1→161 \to 16.19016.1
PG-DPS, w=4w=4.19516.0
Ours.17615.7
UniformfinePG-DPS, scheduled 1→161 \to 16.25024.2
PG-DPS, w=4w=4.28830.7
Ours.14112.6
ZigZagfinePG-DPS, scheduled 1→161 \to 16.29926.9
PG-DPS, w=4w=4.34133.8
Ours.17112.6
GaussianfinePG-DPS, scheduled 1→161 \to 16.31026.1
PG-DPS, w=4w=4.34833.1
Ours.18413.7
mean (9 cells)PG-DPS, scheduled 1→161 \to 16.17420.1
PG-DPS, w=4w=4.19123.4
Ours.11814.0

FFHQ: PG-DPS w/ time-dependent schedule vs Ours

TargetAttributeMethodTV ↓\downarrowFID ↓\downarrow
UniformagePG-DPS, scheduled (best wˉ=8\bar w = 8).017751.98
Ours.02345.18
UniformgenderPG-DPS, scheduled (best wˉ=4\bar w = 4).052148.24
Ours.005244.97
UniformracePG-DPS, scheduled (best wˉ=2\bar w = 2).41954.29
Ours.02847.59
Custom 1agePG-DPS, scheduled (best wˉ=2\bar w = 2).13352.67
Ours.04946.14
Custom 1genderPG-DPS, scheduled (best wˉ=8\bar w = 8).039650.66
Ours.009446.52
Custom 1racePG-DPS, scheduled (best wˉ=4\bar w = 4).31051.38
Ours.04343.03
Custom 2agePG-DPS, scheduled (best wˉ=8\bar w = 8).074755.16
Ours.01346.25
Custom 2genderPG-DPS, scheduled (best wˉ=4\bar w = 4).13555.63
Ours048.69
Custom 2racePG-DPS, scheduled (best wˉ=2\bar w = 2).53253.33
Ours.05346.53
Uniformage ×\times gender ×\times racePG-DPS, scheduled (best wˉ=2\bar w = 2).44449.30
Ours.11241.28
CustomJointage ×\times gender ×\times racePG-DPS, scheduled (best wˉ=8\bar w = 8).34257.46
Ours.14242.19

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal & Updates

We thank the reviewers and the AC for their time. We summarize the discussion and our key updates below.

Key Contributions

Our work introduces inference-time attribute distribution alignment for pretrained unconditional diffusion models. We study how to steer a generated sample population to match flexible test-time attribute target distribution without retraining or target reference samples, where the attribute is induced by a differentiable oracle. We find that most existing guidance methods fall short in better trading-off attribute distribution alignment and sample quality. Our key contributions:

  • Formulation: We propose a stacked dynamical system where batched diffusion/flow dynamics are coupled solely through a batch-level terminal alignment cost and independent modification penalty, capturing population objectives that per-sample rewards cannot express.
  • Algorithm and theory: Terminal-only coupling yields a block-diagonal structure which enables a closed-form controller, supported by formal guarantees on algorithmic stationarity and distribution alignment.
  • Empirical validation: Experiments on image generation spanning diverse targets, joint attributes, and pixel diffusion and latent flow backbones demonstrate stronger alignment while preserving sample quality compared to baselines.

Recognized Strengths

  • Meaningful and well-motivated population-level alignment problem that cannot be expressed via per-sample rewards (gbWg, 86P4, dRf1).
  • Principled modeling that explicitly balances alignment and quality without retraining the base model (gbWg, 86P4, dRf1).
  • Consistent empirical improvements across architectures, target distributions, and joint attributes while preserving fidelity (gbWg, 86P4).
  • Clear presentation (gbWg, 86P4).
  • Method soundness (gbWg, dRf1).

Major Concerns Addressed during Rebuttal

  1. Novelty and positioning:
    • We clarified that our contribution lies in the batch-coupled formulation, terminal-only coupling structure, and setting-specific theoretical results.
    • We clarified the unconditional diffusion generation regime and distinguished our problem scope from the one studied in other reference-based methods and diversity guidance methods without flexible targets.
  2. Baselines and compute:
    • Our method maintains better alignment and quality against retuned and time-scheduled PG in nearly all settings. Against compute-matched unguided and TFG/DPS-guided rejection sampling, joint trajectory optimization consistently preserves higher sample quality across all quality metrics. Additional analyses and a rare-class case study show active steering is essential for rare attributes in the base model’s vanilla generation distribution.
    • We further demonstrated substantial reduction of our method's runtime.
  3. Evaluation reliability and diversity:
    • We added addition cross-model evaluations and classifier confidence margin checks and the results confirm genuine alignment of our method without decision boundary exploitation.
    • We also added no-reference quality and within-group sample diversity evaluations, and these results confirm sample quality and diversity improvements of our method over the baselines.
  4. Technical refinements:
    • We explained and resolved large-batch sensitivity, fixed minor reporting metric inconsistencies, and improved runtime efficiency.

Reviewer Engagement & Points of Discrepancies

  • Reviewer engagement (until 58 mins before the discussion deadline):
    • Reviewer gbWg has not replied to our full rebuttal or follow-up reminder.
    • Reviewer dRf1 responded once and confirmed our oracle-bias resolution, but has not yet replied to our second-round response addressing their remaining points.
    • Reviewer 86P4 did reaffirm their positive assessment after reviewing our responses.
  • Discrepancies:
    • Novelty and hypothetical baseline comparison (dRf1): Reviewer dRf1 interpreted our explanation of why a “baseline” constructed by changing the formulation and the objective is not independent as limited novelty. We clarified that both the stacked batch system and group-level KL objective are our contributions and sharing the pre-existing optimization solver does not diminish our novelty.
    • Alignment–quality trade-off in rejection sampling (dRf1): While Reviewer dRf1 focused solely on alignment error, preserving sample quality is also central to our task, and the guided-rejection experiments in our round-2 response show guidance sacrifices substantial sample quality.
    • Out-of-scope baselines (gbWg): We clarified why comparisons to reference-sample MMD Guidance and the diversity guidance baselines without flexible targets fall outside our scope.

We believe that our responses have answered questions from the reviewers, substantially clarified the main concerns, and strengthened the paper’s technical soundness.

Sincerely,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.