acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Preference Alignment for Diffusion Models through Classifier-Free Guidance

Abstract

Aligning large-scale text-to-image diffusion models with nuanced human preferences remains challenging. While direct preference optimization (DPO) is simple and effective, large-scale finetuning often shows a generalization gap. We take inspiration from test-time guidance and cast preference alignment as classifier-free guidance (CFG): a finetuned preference model acts as an external control signal during sampling. Building on this view, we propose a simple method that improves alignment without retraining the base model. To further enhance generalization, we decouple preference learning into two modules trained on positive and negative data, respectively, and form a contrastive guidance vector at inference by subtracting their predictions (positive minus negative), scaled by a user-chosen strength and added to the base prediction at each step. This yields a sharper and controllable alignment signal. We evaluate on Stable Diffusion 1.5 and Stable Diffusion XL with Pick-a-Pic v2 and HPDv3, showing consistent quantitative and qualitative gains.

Primary area:
generative models
Keywords:
Diffusion model, Preference alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1V8fOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors recast preference alignment as an inference-time guidance problem: instead of deploying the preference-tuned checkpoint, they amplify its prediction residual at sampling time, so that the amount of preference training and the strength of preference guidance can be set separately. They instantiate this idea in two variants. PGD scales the difference between a lightly DPO-tuned model and its reference, while cPGD replaces that residual with the difference between two generators fitted independently to the preferred and dispreferred image marginals under ordinary denoising losses, so that both branches simply learn to generate their own data and repulsion enters only at inference. They evaluate both variants on SD1.5 and SDXL across several prompt suites and reward models, transfer the learned directions to already aligned base models, and assess partial-step guidance and model merging as ways to reduce the added inference cost.

Critical strengths
  1. The paper is clearly written and the method is easy to follow, with the inference rule reducing to a single linear composition of frozen predictors. The training and sampling protocol is documented in enough detail to be reimplemented.
  2. The evaluation is broad, covering two preference datasets, three prompt suites and six reward models, plus diversity and FID. PGD and cPGD improve over the Diffusion-NPO baseline on both backbones, and give nearly consistent gains across several target checkpoints spanning DPO-, MaPO-, KTO- and SPO-tuned SDXL and SD1.5.
Critical weaknesses
  1. Both PGD and cPGD amount to taking the difference between two predictions and adding it to a base prediction at inference, which is already what conditional SFT does by fine-tuning with "good image"/"bad image" prefixes and composing the two conditions through ordinary CFG at sampling time. The novelty is therefore limited: Eqs. 6 and 9 use the same ingredients, with the condition carried by a parameter set rather than a prefix. That change costs three parameter sets and six forward passes per step under the fixed text CFG of Sec. 4.1, whereas CSFT needs a single LoRA that merges back into the base model at no added inference cost.
  2. The construction of D+ and D- for cPGD is never specified, and Pick-a-Pic labels relative preference rather than absolute quality. With A>B and B>C, image B is both a winner and a loser. Eq.5 is per-pair, so Diffusion-DPO never needs the absolute status of B, but Eqs.7-8 fit two marginals and force an assignment that the paper leaves undefined.
Questions for authors
  1. Please add cSFT under the same data and budget, reporting win rates and per-step forward passes beside PGD and cPGD.
  2. Since the method is positioned as inference-time, please also compare at a matched sampling budget against a test-time baseline such as best-of-N selection with the reward models already used here.
  3. Please report the exact rule for building D+ and D-, including how images appearing on both sides are handled and how ties are treated, plus the resulting overlap fraction. Since Pick-a-Pic provides only relative comparisons, it would also help to see the split rebuilt from an absolute quality criterion, for instance by thresholding a reward model score so that the two sets are disjoint by construction, which would indicate how much the present assignment shapes the learned residual.
  4. Since current text-to-image models are predominantly rectified flow architectures, can the learned directions transfer to SD3 or FLUX, where both the latent space and the prediction parameterization differ from SDXL?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the careful review and constructive feedback.

cSFT under the same data and budget, reporting win rates and per-step forward passes beside PGD and cPGD.

We have added this comparison on SD1.5. Conditional SFT (cSFT) trains one model on the same preferred/dispreferred data, with the prefixes "good image:" and "bad image:" as conditions, under a matched optimization-step budget. At inference it composes ϵbad+α(ϵgood−ϵbad)\epsilon_{\mathrm{bad}}+\alpha(\epsilon_{\mathrm{good}}-\epsilon_{\mathrm{bad}}). The last column counts model-branch evaluations per sampling step, before the conditional and unconditional passes of text CFG.

MethodPickScoreWin rate (%)Branch evaluations per step
Base SD1.50.20650.01
cSFT0.20862.32 (good, bad)
PGD0.21378.32 (reference, DPO)
cPGD0.21576.93 (reference, preferred, dispreferred)

Both guidance methods outperform cSFT on both measures. cPGD obtains the highest PickScore and PGD the highest win rate. cPGD uses one more branch evaluation per step than cSFT, and partial-step guidance reduces this cost: guiding only the first 10 of 50 steps takes 1.4× the base sampling time (Table 16). We have added the comparison to Appendix A.3.

Comparison with Best-of-N sampling.

We thank the reviewer for suggesting Best-of-NN as a test-time reference. For each prompt, we generate NN SDXL base samples with the 50-step DDIM sampler and keep the one with the highest PickScore, the same reward used for evaluation.

ConditionPickScoreImprovement over base
Base0.2225–
Best-of-20.2257+0.0033
Best-of-30.2267+0.0043
Full cPGD0.2274+0.0050

Full cPGD reaches a higher PickScore than Best-of-3 although Best-of-NN selects with the evaluation metric itself. The two have the same sampling cost: Best-of-3 runs the base sampler three times, and full cPGD takes 3.0× the base sampling time (Table 16). cPGD also modifies a single trajectory and needs no reward model at inference. We have added the results to Appendix A.7.

Construction of the preferred and dispreferred datasets.

For each preference pair with valid labels and a nonempty caption, we assign the image with the higher preference label to D+\mathcal D_+ and the other image to D−\mathcal D_-, and skip ties. We keep repeated images and pair rows, so an image enters D+\mathcal D_+ once per comparison it wins and D−\mathcal D_- once per comparison it loses. The two subsets are thus the preferred and dispreferred marginals of the pair data (Section 4.2), and an image that wins some comparisons and loses others appears in both. On the Pick-a-Pic v2 training set, 153,624 images appear in both subsets (Table 11). We keep the original pairwise annotations without further filtering. Each subset then fine-tunes its own generator from the same reference checkpoint. We have added the assignment rule and the overlap counts to Appendix A.4.

Transfer between different latent spaces and prediction parameterizations.

Direct transfer requires a shared latent space, matching noise levels, and compatible conditioning (Section 4.1). Prediction parameterization is less restrictive, since noise, score, and velocity predictions are linear in each other at a given noise level (Section 3.1), so the residual can be converted to the target's parameterization. A different latent space requires a representation mapping or a guidance module trained for the target.

Our SDXL-to-KOALA experiment (Figure 17) tests transfer across architectures that share a VAE and latent space. The SDXL guidance modules improve KOALA on PickScore, ImageReward, CLIP, and HPSv2 without updating KOALA's parameters. We have clarified these conditions in Appendix A.5.

Thank you for adding the cSFT comparison and clarifying how the preference datasets were constructed. But I still have some concerns. First, images shared by the preferred and dispreferred subsets may bring the two distributions closer together and weaken the learned guidance signal. The revision reports the extent of the overlap, but it remains unclear how retaining these images affects the learned guidance compared with removing them. Second, I would like to know which dataset the prompts came from and how many were used for the Best-of-3 comparison. If the prompts also came from Pick-a-Pic, it is unclear whether cPGD would still outperform Best-of-3 on a different prompt set or under a different evaluation metric. Third, whether the method offers similar gains on rectified flow models such as SD3 or FLUX remains untested. Due to these concerns, I maintain my original rating.

Reviewer_Z4KWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the limitations of standard preference alignment methods (such as Diffusion-DPO) for text-to-image diffusion models, which often reduce sample diversity or degrade image quality when trained for too long. To overcome this, the authors propose separating the amount of preference training from the strength of preference guidance. Instead of prolonged finetuning, a lightly trained model supplies a reusable guidance direction (residual) whose strength is controlled at inference time. Additionally, the paper introduces Contrastive Preference-Guided Diffusion (cPGD), which independently fits preferred and dispreferred distributions using ordinary denoising losses and contrasts their predictions during sampling to introduce preference guidance with a local connection to reweighted DPO.

Critical strengths

PGD separates the amount of preference training from the strength of preference guidance by using the prediction difference between a lightly DPO-tuned model and its reference during sampling. This is highly consequential because prolonged training in standard DPO can reduce sample diversity and degrade image quality; PGD successfully avoids these issues, yielding favorable reward-diversity and reward-prior-preservation trade-offs while keeping the guidance strength adjustable at inference.

cPGD independently fits preferred and dispreferred distributions using ordinary denoising losses and contrasts their predictions during sampling, establishing a local connection to reweighted DPO. This is consequential because it removes the need for coupled preference losses during training and provides a robust alternative that achieves high performance across standard benchmarks and competitive human evaluations.

Critical weaknesses

The paper does not thoroughly resolve whether introducing a repulsive force against dispreferred generations necessarily results in preferred outputs. As the authors acknowledge, the theoretical connection between the cPGD contrastive update and reweighted DPO is strictly local, and curvature limits this correspondence as parameter displacements grow. Relying solely on repulsion without global theoretical guarantees means that blindly amplifying guidance can magnify errors and distortions alongside useful signals.

In the experimental section, the checkpoints for baseline methods (such as MaPO, NPO, and SPO) are provided by their respective authors rather than being retrained under a single, common training budget and hyperparameter setup. This inconsistency means that performance differences could potentially stem from varying training resources, schedules, or data processing details rather than purely algorithmic advantages.

Questions for authors

The paper notes that the mathematical connection between the cPGD contrastive update and reweighted DPO is strictly local (relying on a single-step gradient analysis and linear approximation). Can the authors clarify to what extent this local approximation breaks down over the multi-step diffusion sampling trajectory due to accumulated errors, and how such error propagation is mitigated in practice?

Future work should consider retraining all baseline comparison methods (such as MaPO, NPO, and SPO) under a strictly unified training budget, dataset, and hyperparameter setup rather than relying on checkpoints provided by respective authors, ensuring that performance gains are purely algorithmic.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the careful review and constructive feedback.

Whether introducing a repulsive force against dispreferred generations necessarily results in preferred outputs?

We agree that a repulsive force against dispreferred generations does not necessarily lead to preferred outputs. Our new branch ablation confirms this. It compares full cPGD with each of its one-sided parts at the same guidance scale (w=10w=10):

  • Positive-only: sref+w(sθ+−sref)s_{\mathrm{ref}}+w(s_{\theta_+}-s_{\mathrm{ref}}), attraction toward the preferred generator only.
  • Negative-only: sref+w(sref−sθ−)s_{\mathrm{ref}}+w(s_{\mathrm{ref}}-s_{\theta_-}), repulsion from the dispreferred generator only.
  • Full cPGD: sref+w(sθ+−sθ−)s_{\mathrm{ref}}+w(s_{\theta_+}-s_{\theta_-}).
BackbonePromptsBasePositive-onlyNegative-onlyFull cPGD
SD1.5Pick-a-Pic v20.20880.19820.16960.2114
SDXLPick-a-Pic v20.22250.20380.17400.2274
SDXLHPDv20.22840.20810.17210.2326

Negative-only guidance lowers PickScore in every setting, and so does positive-only guidance. Only the full direction improves over the base, and on the independent HPDv2 prompts it wins on 71.8% of prompts. The benefit therefore comes from the contrastive composition of the two signals rather than from repulsion alone. We have added this ablation to Appendices A.1 and A.2.

Scope of the theoretical connection.

We agree that the connection between cPGD and preference optimization is not an exact equivalence over an entire finite-step sampling trajectory. Section 4.2 states it as a local, leading-order connection: it holds for one gradient step from a common initialization and relies on linearizing the score predictor (Appendix E.1).

On 80 SDXL prompts with 50 Euler steps, we compare explicit cPGD with a merged predictor that folds the difference of the preferred and dispreferred rank-16 LoRA updates into the base U-Net. Replay relative-L2L_2 error and cosine similarity compare the two predictions along the explicit cPGD trajectory. For final latent drift, the samplers start from the same noise and evolve independently. We average the L2L_2 distance between their final latents divided by the norm of the explicit cPGD final latent plus 10−810^{-8}.

wwReplay relative-L2L_2Replay cosineFinal latent drift
10.0071520.9999660.268822
50.0390770.9989270.470898
100.0833170.9950260.611834
200.1867250.9768210.787439

The two updates agree closely at small ww and diverge gradually as ww grows, with the cosine staying above 0.99 up to w=10w=10. This test characterizes the merged approximation and is not an error bound for the full reverse-sampling process. The theory is thus a local characterization rather than a global finite-step guarantee, and moderate guidance scales and partial-step guidance provide practical ways to manage this trade-off. We have added the test and clarified its scope in Appendix A.6.

Composability with existing preference-optimization methods.

cPGD is designed to complement these training algorithms rather than replace them. It adds a test-time direction to any compatible checkpoint, including checkpoints already tuned by preference optimization. We apply PGD and cPGD to author-released DPO, MaPO, KTO, and SPO checkpoints. Entries are mean win rates (%) against the untuned base over PickScore, HPSv2, HPSv3, Aes, CLIP, and ImageReward, averaged across Pick-a-Pic v2 and Parti-Prompts.

Target checkpointWithout guidance+ PGD+ cPGD
SDXL50.070.370.8
DPO-SDXL65.274.871.2
MaPO-SDXL60.873.671.9
SPO-SDXL76.980.682.2
SD1.550.063.768.8
DPO-SD1.563.866.171.6
KTO-SD1.569.973.470.6
SPO-SD1.561.066.271.7

Both PGD and cPGD improve every target checkpoint, including SPO-SDXL, the strongest released model. The checkpoints come from their authors rather than from one common training budget, so we read these results as evidence of composability. Table 5 in Appendix A collects them from Tables 1 and 2.

Thank you for the detailed responses and additional experiments. The rebuttal addresses some of my concerns and provides useful clarifications.

The authors have provided useful additional experiments and clarifications. The new ablation supports the effectiveness of combining positive and negative guidance, as neither component alone consistently improves preference scores. The trajectory analysis further clarifies the local nature of the theoretical connection to reweighted DPO, but also shows that the approximation error increases with the guidance scale, without providing a global guarantee. Finally, the additional experiments demonstrate that PGD and cPGD can be applied to existing preference-optimized models, but do not fully address the fairness of baseline comparisons, as different training configurations and budgets are still used. Overall, the rebuttal addresses some of my concerns, but several important issues remain. Therefore, I maintain my original rating.

Reviewer_U4cbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper promotes cPGD, a new strategy for preference guidance at the inference stage of T2I diffusion models. Using ordinary denoising losses, cPGD fits two individual generators to the preferred and dispreferred images, with a local connection to reweighted Diffusion-DPO for eventual generation. On both SD1.5 and SDXL backbones, the authors show that cPGD helps achieve higher preference scores in becnhmarks of Pick-a-Pic v2 and Parti-prompts, with better alignments to human verifications.

Critical strengths

I find the following aspects in the paper commendable.

  • cPGD’s diversity and quality improvement from the baseline DPO-approached are visibly tracked alongside the preference scores. The authors have proved the importance of having stronger guidance during inference.
  • Using cPGD, the guidance modules adapt directly from existing checkpoints, skipping the costly re-training part.
Critical weaknesses

The main contribution of this paper is cPGD. Unfortunately, I find the authors have missed out on one critical ablative step.

Headline Weakness - In cPGD, the Contribution of Fitting the Dispreferred Distributions Is Never Singled Out.

According to Section 4.2 and Tables 1 to 3, I find that cPGD is formulated as jointly learning from both the preferred and the dispreferred distributions simultaneously. Since only applying L_pos(theta+) as in Formula 7 is effectively identical with the original PGD, the effect from the actual novel part of cPGD, i.e. steering away from the dispreferred distribution defined by Formula 8, is not ablatively examined in isolation.

Missing this ablative study makes me question the veracity of cPGD. Because both cPGD generators (the positive and the negative) share the same denoising loss and the same base model, we are yet to know if the gains in benchmarking (Tables 1 and 2) may actually be attributed to repulsing the dis-preferred images alone, rather than the bi-directional contrastive preference learning that is cPGD.

My suggestion to fix this is straightforward: Amend Tables 1 and 2 with a dispreferred-only strategy. Given the time limit, I’d like to at least see the comparisons on SD 1.5.

Questions for authors

Please find my suggestion at the end of the Weakness section above. For applying my suggestion, the authors may find the checkpoints from CHATS useful.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the positive review and constructive feedback.

The Contribution of Fitting the Dispreferred Distributions Is Never Singled Out.

We thank the reviewer for this suggestion. To isolate the contribution of the dispreferred distribution, we compare full cPGD with two one-sided variants at the same guidance scale (w=10w=10):

  • Positive-only: sref+w(sθ+−sref)s_{\mathrm{ref}}+w(s_{\theta_+}-s_{\mathrm{ref}}), which removes the dispreferred generator.
  • Negative-only: sref+w(sref−sθ−)s_{\mathrm{ref}}+w(s_{\mathrm{ref}}-s_{\theta_-}), which removes the preferred generator.
  • Full cPGD: sref+w(sθ+−sθ−)s_{\mathrm{ref}}+w(s_{\theta_+}-s_{\theta_-}).

We report PickScore on the Pick-a-Pic v2 test set and, for SDXL, on an independent HPDv2 prompt suite.

BackbonePromptsBasePositive-onlyNegative-onlyFull cPGD
SD1.5Pick-a-Pic v20.20880.19820.16960.2114
SDXLPick-a-Pic v20.22250.20380.17400.2274
SDXLHPDv20.22840.20810.17210.2326

Without the dispreferred generator, guidance lowers PickScore: positive-only guidance falls below the base in all three settings, while full cPGD improves over it. Negative-only guidance falls further, so repulsion alone is also insufficient. The dispreferred distribution is therefore useful through its relative signal with the preferred distribution, likely because both generators share the shift from the pretrained model toward the preference images, and their difference cancels it. We have added this ablation and the definitions to Appendices A.1 and A.2.

Relationship to CHATS.

CHATS [1] is closely related, as both methods model preferred and dispreferred distributions and use both during inference. The two methods differ in training and inference. CHATS jointly trains the two generators with a coupled preference objective and coordinates them through proxy-prompt-based sampling. cPGD fits each distribution independently with the ordinary denoising loss and adds the difference of their predictions to a separate base model through a CFG-like contrastive direction. Because the base model stays separate, the same pair of generators can also guide other compatible checkpoints, including DPO-, MaPO-, KTO-, and SPO-tuned models (Tables 1 and 2). We have expanded this discussion in Appendix A.2, which also evaluates the released CHATS checkpoint.

[1] Minghao Fu, Guo-Hua Wang, Liangfu Cao, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation. ICML, 2025.

I appreciate the authors for running the ablation. That being said, my original request is not fully matched. The new results table only covers PickScore and omits Parti-Prompts. I cannot yet confirm if the dispreferred-only variant falls short across the full board of metrics in Tables 1 and 2. I decide to keep my rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive feedback. We have uploaded a revised paper with new text in blue. All section, figure, and table numbers refer to the revised paper.

In the main Pick-a-Pic v2 benchmark, full cPGD achieves PickScore win rates of 80.0% on SDXL and 76.9% on SD1.5 against the corresponding untuned base models (Tables 1 and 2).

New experiments

  • Branch ablation (Appendices A.1 and A.2). Positive-only and negative-only guidance both lower PickScore on SD1.5 and SDXL, while full cPGD improves over the base. The same ordering holds on an independent HPDv2 prompt suite, where cPGD wins on 71.8% of prompts.
  • Conditional SFT (Appendix A.3). On SD1.5, cSFT reaches a 62.3% PickScore win rate over the base, compared with 76.9% for cPGD and 78.3% for PGD.
  • Approximation across guidance scales (Appendix A.6). Explicit guidance and its merged approximation agree closely at small guidance scales and diverge gradually at larger ones.

Changes to the text

  • Construction of D+\mathcal D_+ and D−\mathcal D_- (Appendix A.4). We state the assignment rule and report how many images appear in both subsets.
  • Relationship to CHATS (Appendix A.2). We expand the comparison of training objectives and sampling, and evaluate the released CHATS checkpoint.
  • Transfer conditions (Appendix A.5). We state when guidance transfers directly across models.
  • Composability (Table 5). We collect the results of applying cPGD to author-released DPO, MaPO, KTO, and SPO checkpoints, where it improves every target.

We hope these additions address the reviewers' concerns and are happy to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.