- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors recast preference alignment as an inference-time guidance problem: instead of deploying the preference-tuned checkpoint, they amplify its prediction residual at sampling time, so that the amount of preference training and the strength of preference guidance can be set separately. They instantiate this idea in two variants. PGD scales the difference between a lightly DPO-tuned model and its reference, while cPGD replaces that residual with the difference between two generators fitted independently to the preferred and dispreferred image marginals under ordinary denoising losses, so that both branches simply learn to generate their own data and repulsion enters only at inference. They evaluate both variants on SD1.5 and SDXL across several prompt suites and reward models, transfer the learned directions to already aligned base models, and assess partial-step guidance and model merging as ways to reduce the added inference cost.
Critical strengths
- The paper is clearly written and the method is easy to follow, with the inference rule reducing to a single linear composition of frozen predictors. The training and sampling protocol is documented in enough detail to be reimplemented.
- The evaluation is broad, covering two preference datasets, three prompt suites and six reward models, plus diversity and FID. PGD and cPGD improve over the Diffusion-NPO baseline on both backbones, and give nearly consistent gains across several target checkpoints spanning DPO-, MaPO-, KTO- and SPO-tuned SDXL and SD1.5.
Critical weaknesses
- Both PGD and cPGD amount to taking the difference between two predictions and adding it to a base prediction at inference, which is already what conditional SFT does by fine-tuning with "good image"/"bad image" prefixes and composing the two conditions through ordinary CFG at sampling time. The novelty is therefore limited: Eqs. 6 and 9 use the same ingredients, with the condition carried by a parameter set rather than a prefix. That change costs three parameter sets and six forward passes per step under the fixed text CFG of Sec. 4.1, whereas CSFT needs a single LoRA that merges back into the base model at no added inference cost.
- The construction of D+ and D- for cPGD is never specified, and Pick-a-Pic labels relative preference rather than absolute quality. With A>B and B>C, image B is both a winner and a loser. Eq.5 is per-pair, so Diffusion-DPO never needs the absolute status of B, but Eqs.7-8 fit two marginals and force an assignment that the paper leaves undefined.
Questions for authors
- Please add cSFT under the same data and budget, reporting win rates and per-step forward passes beside PGD and cPGD.
- Since the method is positioned as inference-time, please also compare at a matched sampling budget against a test-time baseline such as best-of-N selection with the reward models already used here.
- Please report the exact rule for building D+ and D-, including how images appearing on both sides are handled and how ties are treated, plus the resulting overlap fraction. Since Pick-a-Pic provides only relative comparisons, it would also help to see the split rebuilt from an absolute quality criterion, for instance by thresholding a reward model score so that the two sets are disjoint by construction, which would indicate how much the present assignment shapes the learned residual.
- Since current text-to-image models are predominantly rectified flow architectures, can the learned directions transfer to SD3 or FLUX, where both the latent space and the prediction parameterization differ from SDXL?
2 comments
Thank you for adding the cSFT comparison and clarifying how the preference datasets were constructed. But I still have some concerns. First, images shared by the preferred and dispreferred subsets may bring the two distributions closer together and weaken the learned guidance signal. The revision reports the extent of the overlap, but it remains unclear how retaining these images affects the learned guidance compared with removing them. Second, I would like to know which dataset the prompts came from and how many were used for the Best-of-3 comparison. If the prompts also came from Pick-a-Pic, it is unclear whether cPGD would still outperform Best-of-3 on a different prompt set or under a different evaluation metric. Third, whether the method offers similar gains on rectified flow models such as SD3 or FLUX remains untested. Due to these concerns, I maintain my original rating.
Official Comment by Authors
We thank the reviewer for the careful review and constructive feedback.
We have added this comparison on SD1.5. Conditional SFT (cSFT) trains one model on the same preferred/dispreferred data, with the prefixes "good image:" and "bad image:" as conditions, under a matched optimization-step budget. At inference it composes ϵbad+α(ϵgood−ϵbad). The last column counts model-branch evaluations per sampling step, before the conditional and unconditional passes of text CFG.
Both guidance methods outperform cSFT on both measures. cPGD obtains the highest PickScore and PGD the highest win rate. cPGD uses one more branch evaluation per step than cSFT, and partial-step guidance reduces this cost: guiding only the first 10 of 50 steps takes 1.4× the base sampling time (Table 16). We have added the comparison to Appendix A.3.
We thank the reviewer for suggesting Best-of-N as a test-time reference. For each prompt, we generate N SDXL base samples with the 50-step DDIM sampler and keep the one with the highest PickScore, the same reward used for evaluation.
Full cPGD reaches a higher PickScore than Best-of-3 although Best-of-N selects with the evaluation metric itself. The two have the same sampling cost: Best-of-3 runs the base sampler three times, and full cPGD takes 3.0× the base sampling time (Table 16). cPGD also modifies a single trajectory and needs no reward model at inference. We have added the results to Appendix A.7.
For each preference pair with valid labels and a nonempty caption, we assign the image with the higher preference label to D+ and the other image to D−, and skip ties. We keep repeated images and pair rows, so an image enters D+ once per comparison it wins and D− once per comparison it loses. The two subsets are thus the preferred and dispreferred marginals of the pair data (Section 4.2), and an image that wins some comparisons and loses others appears in both. On the Pick-a-Pic v2 training set, 153,624 images appear in both subsets (Table 11). We keep the original pairwise annotations without further filtering. Each subset then fine-tunes its own generator from the same reference checkpoint. We have added the assignment rule and the overlap counts to Appendix A.4.
Direct transfer requires a shared latent space, matching noise levels, and compatible conditioning (Section 4.1). Prediction parameterization is less restrictive, since noise, score, and velocity predictions are linear in each other at a given noise level (Section 3.1), so the residual can be converted to the target's parameterization. A different latent space requires a representation mapping or a guidance module trained for the target.
Our SDXL-to-KOALA experiment (Figure 17) tests transfer across architectures that share a VAE and latent space. The SDXL guidance modules improve KOALA on PickScore, ImageReward, CLIP, and HPSv2 without updating KOALA's parameters. We have clarified these conditions in Appendix A.5.