acceptodds
Under review as a conference paper at ICLR 2027

FestDPO: Few-step Generator Alignment with Direct Preference Optimization

Abstract

Few-step generative models can generate high-fidelity samples within a few function evaluations. Despite this efficiency, generated samples may not exhibit desirable properties. When these properties are difficult to encode as an explicit reward function, direct preference optimization (DPO) can align generative models using pairwise preference feedback without training a separate reward model. However, extending DPO to few-step generative models is challenging because few-step generative models are generally implicit, making the likelihood evaluation required by DPO intractable. To address this challenge, we introduce Few-step DPO (FestDPO), an extension of DPO for few-step generative models that leverages nonparametric likelihood estimation from empirical samples. By exploiting the fast sampling capabilities of few-step generative models, our approach makes sample-based approximation of DPO loss computationally feasible. Furthermore, the sample-based formulation makes FestDPO agnostic to the model family and sampling procedure. Our toy experiment demonstrates that FestDPO matches the reward-tilted target distribution across four few-step generators. For real-world tasks, we evaluate FestDPO in two domains: text-to-image generation and protein backbone generation. In text-to-image generation, FestDPO outperforms preference optimization baselines in both win rates against the base models and human evaluation scores. In protein backbone generation, it achieves a higher -sheet fraction and better structural designability than the baselines.

Primary area:
reinforcement learning
Keywords:
Preference Optimization, Few-step Generative Models, Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GYgkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present an approach (FestDPO) to apply direct preference optimization to few-step generative models where the likelihoods are implicit. They address this issue by using Kernel Density Estimation (KDE) to estimate the likelihoods using multiple samples generated by the frozen reference model and the model with the weights being fine-tuned. They argue that this is a tractable approach since few-step models are naturally very fast at generating samples. They present theoretical analysis showing that the FestDPO objective approaches that of DPO as the number of samples used for KDE grows. Finally they compare the effectiveness of FestDPO empirically to two recent baselines across text-to-image and protein generation modalities, showing that FestDPO leads to improved performance in terms of PickScore, Aesthetic score, and human evaluation.

Critical strengths

1- The central idea is simple, well-presented and motivated, and effective.

2- Experiments across multiple modalities show FestDPO seems promising (even though the improvements at times seem marginal e.g. Table 10)

3- well-written, easy to follow, good figures. Also includes ablations that show the robustness of the approach across N, and different encoders.

Critical weaknesses

1- I am unsure of the novelty/significance of the approach. Conceptually, FestDPO's final proposed objective seems closely related to that of DRPO as it also uses contrastive reference pairs to construct its fine-tuning objective. Furthermore, the idea of using KDE to estimate the score for implicit few step models appears in [1]. In my opinion, a more convincing argument regarding the novelty and significance of FestDPO is missing in the paper.

2- In the qualitative comparisons in the appendix (Figures 8 and 9) the two baselines (especially DrPO) seem nearly identical to the Base model. This raises concerns for me regarding whether the implementation/hyperparameters used for the baseline results were tuned properly.

3- There is a lack of discussion regarding sample complexity of the KDE, Especially as in high dimensions, we would expect to need many samples. The ablation in the experiments addresses this in somewhat in practice, but more thorough investigation and discussion, especially on the theory side would be helpful. Also, are the theoretical results/sample complexity affected by choice of pertained encoders? If so, how?

If these weaknesses are sufficiently addressed, I am open to raising my score.

Questions for authors

1- FestDPO’s final objective appears closely related to DRPO’s, since both build the fine-tuning objective from contrastive reference pairs. What is the conceptual difference between the two objectives, and in what settings would FestDPO be expected to succeed where DRPO fails?

2- To my understanding, using KDE to estimate the score for implicit few-step models has been proposed in [1] (as the authors of the current paper also cite them). Beyond combining this with preference optimization, is there anything new in how FestDPO uses KDE? Could the authors add a direct comparison or discussion to clarify the contribution?

3- In the qualitative comparisons (Figures 8 and 9), the baselines, especially DRPO, look nearly identical to the base model. How were the baseline implementations and hyperparameters chosen and tuned? Could the authors report a hyperparameter sweep or other evidence that the baselines actually moved away from the base model, for example significant reward difference or KL divergence relative to the base model?

4- KDE is known to need many samples in high dimensions. What are the sample complexity requirements of the KDE-based score estimate in this setting, and can the authors provide theoretical bounds or at least a clear discussion of how estimation error scales with dimension and number of samples? Furthermore, are the theoretical results and the sample complexity affected by the choice of pretrained encoder, for example through its embedding dimension or the geometry of its feature space? (the empirical ablations already somewhat address this concern empirically, I'm mainly interested in a theoretical or intuitive explanation)

6 comments

Thank you for your thoughtful review!

Before addressing your concerns, I would like to clarify one point. Could you please specify which paper you are referring to as [1]?

Yes, my apologies, I should have included this in the review:

[1]: Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inference (Lee et. al.)

W1 & Q2) Novelty and significance of FestDPO

Beyond combining this with preference optimization, is there anything new in how FestDPO uses KDE?

What is the conceptual difference between the two objectives, and in what settings would FestDPO be expected to succeed where DRPO fails?

Our core contribution is a sample-based DPO formulation for few-step generative models using non-parametric likelihood estimation, with an asymptotic connection to the reward-tilted distribution induced by the Bradley–Terry model. This contribution offers distinct capabilities and novelty beyond FAV and DrPO.

Firstly, FestDPO can learn directly from preference pairs without reward evaluations or gradients, while FAV requires an explicit differentiable reward. This property of FestDPO avoids fitting a separate reward model, which can require substantial data and careful tuning [2,3,4], Therefore, FestDPO is applicable when preferences are available, but a reliable reward model is difficult to obtain.

Second, DrPO does not follow the Bradley–Terry preference model and shows limited improvement in our offline experiments. The target distribution of DrPO is defined by a contrastive energy E(x)E(x) and does not generally coincide with the reward r(x)r(x) that determines the preference. Therefore, DrPO cannot proportionally sample from p(x)exp⁡(r(x))p(x)\exp(r(x)) while FestDPO can. Empirically, in the 1-D toy setting, DrPO fails to sample in proportion to p(x)exp⁡(r(x))p(x)\exp(r(x)) despite an extensive hyperparameter search. These results are shown in Figure 7 of the appendix. In image and protein tasks, FestDPO significantly outperforms DrPO in the offline setting, where no additional reward queries are available during training. This is particularly important in the standard DPO setting, where offline preferences are available but a reliable reward function is inaccessible.

[2] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." NeurIPS 2023: 53728-53741.

[3] An, Gaon, et al. "Direct preference-based policy optimization without reward modeling." NeurIPS 2023: 70247-70266.

[4] Zhang, Yang, and Jeffrey Skolnick. "Scoring function for automated assessment of protein structure template quality." Proteins: Structure, Function, and Bioinformatics 57.4 (2004): 702-710.

W2 & Q3) Faithfulness of baseline reproduction

How were the baseline implementations and hyperparameters chosen and tuned?

For each baseline result of the paper (PSO and DrPO), we conducted a 4×44\times4 grid search based on the hyperparameters recommended in the original paper and selected the best-performing configuration. As demonstrated in Table G, DrPO does not improve its reward in the offline setting over grid search. For PSO, we selected the indicated best-performing combination of β=10\beta=10 and LR=10−4LR=10^{-4}. We did not undertune the baselines to favor our method.

Table G: Hyperparameter tuning of DrPO on SDXL-DMD2 (4 seeds).

PickScoreLR=10−5LR=10^{-5}LR=5×10−5LR=5\times10^{-5}LR=10−4LR=10^{-4}LR=5×10−4LR=5\times10^{-4}
λ\lambda=0.10.21170.21170.21170.2114
λ\lambda=0.20.21130.21120.21140.2111
λ\lambda=0.30.21090.21070.21100.2120
λ\lambda=0.50.21070.21070.21080.2107

Table H: Hyperparameter tuning of PSO on SDXL-DMD2 (4 seeds).

PickScoreLR=10−5LR=10^{-5}LR=5×10−5LR=5\times10^{-5}LR=10−4LR=10^{-4}LR=5×10−4LR=5\times10^{-4}
β\beta=10.21380.21070.21420.1844
β\beta=100.21370.21440.21580.2087
β\beta=500.21150.21200.21230.2133
β\beta=1000.21120.21140.21160.2123

W3 & Q4) Sample complexity of KDE

How estimation error scales with dimension and number of samples?

Are the theoretical results/sample complexity affected by choice of pertained encoders?

For a fixed bandwidth hh, both the KDE density π^θ\hat{\pi}_{\theta} and its gradient can be expressed as Monte Carlo sample averages. Under standard regularity conditions, both estimators exhibit the following convergence rates [5,6]:

π^θ(y)−πθ(y)=Op(h2+1Nhd)\hat{\pi}_{\theta}(y) - \pi _\theta(y)=O_p\left(h^2+\frac{1}{\sqrt{Nh^{d}}}\right)

∣∇yπ^θ(y)−∇yπθ(y)∣=Op(h2+1Nhd+2)|\nabla_y\hat{\pi}_{\theta}(y) - \nabla_y\pi _\theta(y)|=O_p\left(h^2+\frac{1}{\sqrt{Nh^{d+2}}}\right)

The score estimator ∇log⁡π^θ(y)\nabla \log\hat\pi_\theta(y) inherits the convergence rate of the density estimator. Although KDE in feature space may lose information from the raw space, a pretrained encoder can reduce the dimensionality of the space, mitigating the sample complexity of KDE. The choice of encoder affects the embedding dimension and feature-space geometry, but our ablation study shows that FestDPO remains robust across different pretrained encoders, including Latent-MAE, CLIP, and DINOv2.

[5] Mohamed, Shakir, et al. "Monte carlo gradient estimation in machine learning." JMLR 21.132 (2020): 1-62.

[6] Ascoli, Ruben G. "Limitations of richardson extrapolation for kernel density estimation." arXiv preprint arXiv:1812.08619 (2018).

Reviewer_QuSwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FestDPO, a preference-optimization method for implicit few-step generators whose likelihoods are intractable. It approximates the required density terms using kernel density estimation (KDE) in a pretrained feature space and provides asymptotic consistency results. Experiments cover synthetic distributions, text-to-image generation, and protein backbone generation. Although the approach is promising, the practical reliability of its density estimates and the strength of the evidence supporting its effectiveness at realistic sample sizes remain insufficiently established.

Critical strengths
  • The paper addresses a meaningful limitation of direct preference optimization when applied to implicit generators without tractable likelihoods.

  • The method is evaluated across several domains, and the theoretical analysis provides useful motivation for the proposed estimator.

  • The results suggest that the method can improve selected preference-alignment and protein-design metrics.

Critical weaknesses
  • The gap between asymptotic consistency and practical performance is not adequately resolved. KDE consistency in the asymptotic regime does not guarantee reliable estimates or useful gradients at the finite sample sizes available during training, particularly in high-dimensional feature spaces. This is a central concern because the quality of the approximation directly affects the optimization objective.

  • The method's sensitivity to estimator design is insufficiently characterized. The approach depends on the pretrained feature representation and KDE configuration. More systematic analysis of bandwidth selection, sample count, and feature-space choice is needed to establish that the reported gains are robust rather than configuration-dependent.

  • The computational trade-off is unclear. The paper should provide stronger evidence about training cost, sampling overhead, and wall-clock efficiency compared with relevant baselines under matched compute budgets. These costs are important when evaluating the practical value of the method.

  • Some reported gains may come with quality trade-offs. In the protein-generation experiments, improvements in selected metrics are accompanied by reduced diversity relative to the base model. The paper should clarify whether the overall quality improvement persists when diversity and other relevant measures are considered jointly.

Questions for authors
  1. How sensitive are the results to the number of samples used for KDE and to bandwidth selection? Do the improvements persist at realistic training-time sample budgets?

  2. What evidence demonstrates that the estimated objective and gradients are sufficiently reliable at finite sample sizes, beyond the asymptotic consistency analysis?

  3. How does FestDPO compare with relevant baselines in wall-clock training time and total sampling or compute cost under matched budgets?

  4. For protein generation, do the improvements remain meaningful when diversity is evaluated jointly with β-sheet fraction and structural quality?

5 comments

W1 & Q1) Reliability of estimated gradient

What evidence demonstrates that the estimated objective and gradients are sufficiently reliable at finite sample sizes, beyond the asymptotic consistency analysis?

We agree that asymptotic consistency alone does not establish the reliability of finite-sample estimates. To provide insight about the reliability of the gradient, we measure the cosine similarity of the gradient over all 23.2M trainable LoRA parameters in SDXL-DMD2, holding the preference pair fixed. For each N=1N=1 to 6464, we partition a pool of 2048 samples into ⌊2048/N⌋\lfloor 2048/N\rfloor non-overlapping groups and average gradients from these sets. We then compare this average with the reference gradient computed from all 2048 samples. Table A shows that cosine similarity increases from 0.724 at N=1N=1 to 0.834 at our default N=12N=12, reaching 0.910 at N=64N=64. These results show that the averaged NN-sample gradients align well with the large-sample reference, including at our default N=12N=12.

Table A. Gradient similarity with N=2048N=2048 across different sample counts

NNCosine Sim.
10.724
20.748
40.792
80.811
120.834
160.839
240.862
320.871
640.910

Table B shows that both rewards decline during training with N=1N=1 but improve consistently with our default N=12N=12. This complements Table A by demonstrating that a practical sample size provides useful gradients for optimization.

Table B: Alignment performance of N=12N=12 and N=1N=1

StepPickScore (N=12N=12) ↑PickScore (N=1N=1) ↑Aesthetic (N=12N=12) ↑Aesthetic (N=1N=1) ↑
0 (Base)0.210650.210655.3745.374
1000.216660.209635.7395.272
2000.219480.209165.8475.243
3000.219790.208805.8675.226

W2 & Q1) Sensitivity Analysis

More systematic analysis of bandwidth selection, sample count, and feature-space choice is needed to establish that the reported gains are robust rather than configuration-dependent.

We kindly inform to reviewer that the sensitivity result of sample count and feature encoder has already been evaluated in Tables 1 and 2, respectively. FestDPO maintains stable performance with 8–24 samples within approximately 200 GPU-hours on an RTX 3090. FestDPO is robust to the choice of pretrained feature encoder, with Latent-MAE, CLIP, and DINOv2.

FestDPO uses two kernels with bandwidths, (h1,h2)(h_1,h_2) where h2=h1+δh_2=h_1+\delta. We evaluate bandwidth sensitivity on SDXL-DMD2 using PickScore, with a base-model score of 0.2107. Table C shows that all tested configurations improve upon the base model, demonstrating the robustness of FestDPO.

Table C: Sensitivity analysis of KDE bandwidth (4 seeds)

PickScoreh1h_1=0.2h1h_1=0.25
δ\delta=0.50.21380.2157
δ\delta=0.10.2198 (default)0.2141
δ\delta=1.50.21290.2166

To reduce hyperparameter search costs, we use the median heuristic [1] to adaptively set a single bandwidth to the median pairwise distance between embeddings within each batch. This yields a PickScore of 0.2155, with an average bandwidth of hh=0.175. Although slightly below the default configuration, this result suggests a reasonable starting point for bandwidth tuning. We will include these results in the revised manuscript.

[1] Garreau, Damien, Wittawat Jitkrittum, and Motonobu Kanagawa. "Large sample analysis of the median heuristic." arXiv preprint arXiv:1707.07269 (2017).

W3 & Q3) Computational efficiency under matched budgets

How does FestDPO compare with relevant baselines in wall-clock training time and total sampling or compute cost under matched budgets?

Table D shows the optimization progress of FestDPO and the baselines under the same wall-clock time. As shown below, FestDPO consistently outperforms DrPO. Although FestDPO underperforms PSO through four hours, FestDPO continues to improve while PSO plateaus, surpassing PSO from eight hours onward.

Table D. Anytime performance across wall-clock time (PickScore, 4 seeds)

Prompt setMethod2 hrs4 hrs8 hrs16 hrs
Pick-a-Pic v2SDXL-DMD2 (base)21.0621.0621.0621.06
DrPO21.0721.0721.0921.12
PSO21.4521.5121.5821.58
FestDPO (Ours)21.1821.4821.8521.97
Parti-PromptsSDXL-DMD2 (base)21.6521.6521.6521.65
DrPO21.6521.6521.6621.68
PSO21.9322.0222.0622.06
FestDPO (Ours)21.7521.9722.3022.43
HPSv2SDXL-DMD2 (base)21.4521.4521.4521.45
DrPO21.4621.4621.4921.52
PSO21.9322.0022.0522.05
FestDPO (Ours)21.6021.9422.2922.42

Table E shows that FestDPO underperforms the baselines at training-time sampling budgets up to 245k samples. However, FestDPO is designed to approximate DPO using many inexpensive samples from a few-step generator, making equal sampling budgets unfavorable to FestDPO.

Table E. Anytime performance across the number of samples during training-time (PickScore, 4 seeds)

Prompt setMethod≈122k≈245k≈368k
Pick-a-Pic v2SDXL-DMD2 (base)21.0621.0621.06
PSO21.3521.3421.39
DrPO21.0721.0721.09
FestDPO (Ours)21.1021.2921.48
Parti-PromptsSDXL-DMD2 (base)21.6521.6521.65
PSO21.8421.8821.88
DrPO21.6521.6521.66
FestDPO (Ours)21.6821.8421.97
HPSv2SDXL-DMD2 (base)21.4521.4521.45
PSO21.8421.8721.86
DrPO21.4621.4721.49
FestDPO (Ours)21.5321.7521.94

W4 & Q4) Diversity on image and protein generation

The paper should clarify whether the overall quality improvement persists when diversity and other relevant measures are considered jointly.

For protein generation, do the improvements remain meaningful when diversity is evaluated jointly with β-sheet fraction and structural quality?

To assess image diversity, we report average pairwise LPIPS and DreamSim distances following [2, 3]. In Table F, FestDPO has lower diversity than DrPO but achieves substantially greater gains in the target reward, whereas DrPO improves reward marginally. PSO achieves smaller reward gains than FestDPO while exhibiting a larger decline in diversity. These results show that FestDPO can optimize target reward with a better reward-diversity trade-off than PSO. For qualitative analysis, we provide images with the same prompt generated from FestDPO in Appendix H.

Table F: Alignment performance and diversity across prompt sets (4 seeds)

Prompt setMethodPickScore ↑LPIPS ↑DreamSim ↑
Pick-a-Pic v2SDXL-Turbo (base)22.370.62680.1845
PSO22.460.61040.1680
DrPO22.380.63520.1926
FestDPO (ours)22.600.61610.1718
Parti-PromptsSDXL-Turbo (base)22.780.62430.1906
PSO22.860.61100.1761
DrPO22.780.63420.1983
FestDPO (ours)22.920.61600.1808

For protein generation, Tables 4 and 5 in the paper show that FestDPO reduces diversity relative to the base model but achieves both higher rewards and greater diversity than DrPO. Figure 6 provides qualitative examples with varied secondary structures generated by FestDPO.

[2] Kim, Sunwoo, Minkyu Kim, and Dongmin Park. "Test-time alignment of diffusion models without reward over-optimization." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Kang, Hyeongyu, et al. "Diffusion fine-tuning via reparameterized policy gradient of the soft q-function." International Conference on Learning Representations. Vol. 2026. 2026.

Thank you again for your detailed review and valuable suggestions. We hope our responses have addressed your concerns. Please let us know if you have any further questions.

Reviewer_jA8UOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce Few-step DPO (FestDPO), an extension of DPO for few-step generative models that leverages nonparametric likelihood estimation from empirical samples

Critical strengths

The paper identifies a real and clearly articulated obstacle: few-step generators are implicit models whose likelihoods cannot be evaluated, so DPO cannot be applied directly. The proposed solution is simple and relies on neither denoising trajectories nor model-family-specific derivations. The writing is clear and the structure is clean. The method yields consistent win-rate improvements across the three task families evaluated.

Critical weaknesses
  1. Equation (11) estimates densities of encoded outputs rather than densities of the original generator. For a non-injective encoder, feature-space likelihood ratios need not preserve the raw-space ratios required by DPO, even with infinitely many samples. Different output distributions can induce the same feature distribution while differing in preference-relevant details and their divergence from the reference. Moreover, with a fixed bandwidth, the estimator targets smoothed feature distributions, introducing a further discrepancy that does not disappear merely by increasing the sample count. The paper establishes neither conditions under which these substitutions preserve the DPO objective nor evidence that such conditions hold in its applications. This is a structural gap in the central methodological claim

  2. Proposition 1 follows from standard KDE consistency, continuous-mapping arguments, and uniform integrability. Proposition 2 applies a standard argmin-consistency result, explicitly assuming uniform convergence over the parameter space and a unique, well-separated minimizer. These assumptions are not established for the neural generators or the implemented training objective. Neither proposition characterizes finite-sample log-ratio error, gradient error, fixed-bandwidth bias, or the consequences of feature encoding. Consequently, the analysis provides a conditional asymptotic justification for an idealized estimator, but little insight into the difficult regime introduced by the proposed method.

  3. The only direct demonstration of target-distribution recovery is a one-dimensional toy experiment using 2,048–8,192 KDE samples (Table 7). In contrast, the image experiments use 12 samples, and the protein experiments use 64 samples in a 128-dimensional feature space. No controlled experiment bridges these markedly different estimation regimes. In particular, the paper does not measure the bias or variability of the estimated log-ratios, preference margins, or optimization gradients, nor examine how these quantities depend on bandwidth and effective feature dimension.

  4. Varying the sample count between 8, 12, and 24 and exchanging pretrained encoders does not determine whether KDE aggregation is necessary for the gains. For a Gaussian kernel, the single-sample version already reduces to a simple feature-space logistic preference objective, making an N=1 control particularly informative. Matched nearest-neighbor or direct feature-distance alternatives would further help distinguish the benefit of density aggregation from that of the underlying attraction–repulsion update. Although DrPO is a relevant baseline and uses the same image encoder, comparing two complete algorithms does not isolate the contribution of KDE aggregation or reference normalization within FestDPO. Without such controls, the experiments leave unresolved whether the central methodological addition is responsible for the improvement, or whether a substantially simpler preference loss would perform similarly.

  5. The paper motivates FestDPO through the low sampling cost of few-step generators, but its image objective requires 12 samples from each of the trainable and reference generators per prompt, together with feature extraction through a 340M-parameter encoder and differentiation through the generated features. Nevertheless, the “anytime performance” comparison in Figure 5 is indexed by optimization steps rather than elapsed time or computational budget. The NFE=1 versus NFE=2 experiment establishes applicability at two sampling settings, but does not characterize the tradeoff between sampling cost, estimation quality, and alignment performance.

  6. The evaluation focuses primarily on preference scores, while providing limited evidence about the accompanying changes in diversity and coverage. The protein results already show reduced diversity relative to the base model, which the authors acknowledge, and no corresponding image diversity or coverage evaluation is reported.

Questions for authors

The ablations vary the number of samples, the feature encoder, but none of them isolates the kernel aggregation mechanism itself. Given that increasing N from 8 to 24 barely changes the results, does the improvement genuinely stem from density estimation, or is it closer to an attraction–repulsion optimization in feature space?

The theory and the algorithm operate in the raw space and the feature space respectively; should the discrepancy between them — the compression loss — be quantified?

In high-dimensional settings, is the use of a pretrained encoder unavoidable?

7 comments

W1-1 & Q2) Discrepancy between raw-space objective and feature-space objective

For a non-injective encoder, feature-space likelihood ratios need not preserve the raw-space ratios required by DPO, even with infinitely many samples.

The theory and the algorithm operate in the raw space and the feature space respectively; should the discrepancy between them — the compression loss — be quantified?

Thank you for raising this issue. We agree that a non-injective encoder need not preserve raw-space likelihood ratios, even with infinitely many samples. Equation (11) should therefore be interpreted as a feature-space surrogate. We revise our paper to clarify that our objective in feature space is a surrogate objective. Regarding Q2, raw-space likelihood ratios are intractable for our implicit generators, preventing direct evaluation of the exact discrepancy you required. We would welcome suggestions for reliably quantifying this discrepancy in high-dimensional implicit models and would be happy to explore them.

W1-2 & W2-1) Scope of the asymptotic guarantees

W1-2) Moreover, with a fixed bandwidth, the estimator targets smoothed feature distributions, introducing a further discrepancy that does not disappear merely by increasing the sample count. The paper establishes neither conditions under which these substitutions preserve the DPO objective nor evidence that such conditions hold in its applications.

W2-1) Assumptions are not established for the neural generators or the implemented training objective. ... Consequently, the analysis provides a conditional asymptotic justification for an idealized estimator, but little insight into the difficult regime introduced by the proposed method.

We kindly ask the reviewer to distinguish the scope of our theoretical analysis from the scope of our experiments. Our asymptotic analysis establishes a connection between FestDPO and DPO under idealized conditions (e.g., h→0, Nhd→∞h\rightarrow 0,~Nh^d\rightarrow \infty, and a well-separated minimizer). We do not claim that asymptotic guarantees of FestDPO apply to our applications that use neural networks. Complementing the asymptotic analysis, our toy setting demonstrates that FestDPO can approximate a target distribution using a finite sample and small bandwidths with a neural network. For high-dimensional image and protein tasks, it is ture that we cannot satisfy the asymptotic condition of FestDPO for unbiased estimation. However, we demonstrate practical advantage and effectiveness of FestDPO in diverse domains.

W2-2) Further theoretical analysis is required.

Neither proposition characterizes finite-sample log-ratio error, gradient error, fixed-bandwidth bias, ...

To address your concern, we expand our theoretical analysis of finite-sample log-ratio bias and fixed-bandwidth bias. For a fixed bandwidth hh, the estimated log-ratio has bias O(N−1)O(N^{-1}). For a Gaussian kernel and sufficiently smooth, positive densities, the fixed-bandwidth bias of the log-ratio is O(h2)O(h^2). Under the corresponding regularity conditions, the log-ratio estimation error vanishes as h→0h\to0 and Nhd→∞Nh^d\to\infty, consistent with our asymptotic analysis. We provide the detailed assumptions and derivations of our further analysis in Appendix D.

W5) Computational efficiency of FestDPO

Nevertheless, the “anytime performance” comparison in Figure 5 is indexed by optimization steps rather than elapsed time or computational budget.

We agree that the anytime performance comparison does not account for the tradeoff between sampling cost and alignment performance. To account for this, in Table D, we compare FestDPO with the baselines under equal wall-clock time budgets using SDXL-DMD2. FestDPO consistently outperforms DrPO, and although FestDPO underperforms PSO for the initial four hours, FestDPO continues to improve while PSO plateaus. We will include a wall-clock comparison in the revised manuscript.

Table D. Anytime performance across wall-clock time (PickScore, 4 seeds)

Prompt setMethod2 hrs4 hrs8 hrs16 hrs
Pick-a-Pic v2DrPO21.0721.0721.0921.12
PSO21.4521.5121.5821.58
FestDPO21.1821.4821.8521.97
Parti-PromptsDrPO21.6521.6521.6621.68
PSO21.9322.0222.0622.06
FestDPO21.7521.9722.3022.43
HPSv2DrPO21.4621.4621.4921.52
PSO21.9322.0022.0522.05
FestDPO21.6021.9422.2922.42

W3) Finite-sample estimation error in high dimensions

In particular, the paper does not measure the bias or variability of the estimated log-ratios, preference margins, or optimization gradients, nor examine how these quantities depend on bandwidth and effective feature dimension.

We investigate the finite-sample estimation error in high-dimensional feature spaces by varying the KDE samples NN and bandwidth hh and encoder ϕ\phi. We estimate the log-ratio R=log⁡π^θ(y∣x)π^ref(y∣x)R =\log \frac{\hat \pi_\theta (y|x)}{\hat \pi_{\text{ref}} (y|x)} and the preference margin Rw−RlR_w-R_l, and then compare these with the reference R2048R_{2048} computed from the N=2048N=2048 setup. In general, KDE exhibits a bias-variance trade-off, with variance scaling as O(1/(Nhd))O(1/(Nh^d)) (see Appendix D).

Tables A and B show that increasing NN from 12 to 24 consistently reduces the variance of estimated log-ratios, while the estimated bias remains relatively stable across NN. Increasing the bandwidth hh further reduces estimation variance, although it may introduce additional smoothing bias.

Table A: Finite-Sample log-ratio estimation with Latent-MAE

Metrich=0.1h=0.1h=0.2h=0.2h=0.3h=0.3h=0.4h=0.4h=0.5h=0.5h=1h=1h=2h=2
N=12N=12∣E[R]−R2048∣\lvert\mathbb{E}[R]-R_{2048}\rvert0.07570.00710.00690.01180.01410.01720.0180
Rw−RlR_w-R_l-0.1624-0.0416-0.0186-0.0105-0.0067-0.0017-0.0004
Var[R]\mathrm{Var}[R]0.01260.00090.00020.00010.00000.00000.0000
N=24N=24∣E[R]−R2048∣\lvert\mathbb{E}[R]-R_{2048}\rvert0.07530.00700.00690.01180.01410.01720.0180
Rw−RlR_w-R_l-0.1616-0.0416-0.0186-0.0105-0.0067-0.0017-0.0004
Var[R]\mathrm{Var}[R]0.00660.00050.00010.00000.00000.00000.0000

Table B: Finite-Sample log-ratio estimation with CLIP

CLIPMetrich=0.1h=0.1h=0.2h=0.2h=0.3h=0.3h=0.4h=0.4h=0.5h=0.5h=1h=1h=2h=2
N=12N=12∣E[R]−R2048∣\mid E[R]-R_{2048}\mid0.16820.02600.02300.04300.05280.06650.0700
Rw−RlR_w-R_l-0.0506-0.0063-0.0022-0.0011-0.0007-0.0002-0.0000
Var[R]Var[R]0.47560.01680.00330.00110.00040.00000.0000
N=24N=24∣E[R]−R2048∣\mid E[R]-R_{2048}\mid0.14150.02500.02320.04300.05280.06650.0700
Rw−RlR_w-R_l-0.0569-0.0064-0.0022-0.0011-0.0007-0.0002-0.0000
Var[R]Var[R]0.28220.00860.00170.00060.00020.00000.0000

W4 & Q1) Necessity of KDE aggregation

Given that increasing N from 8 to 24 barely changes the results, does the improvement genuinely stem from density estimation, or is it closer to an attraction–repulsion optimization in feature space?

Thank you for suggesting an informative ablation setup. As you recommended, we assess the necessity of the KDE by setting N=1N=1, making the objective a logistic preference objective. Table C shows an impressive result: by setting N=1N=1, the performance target score gets worse as the step progresses for both reward functions. This result shows that a simpler logistic objective is not enough, and we need to construct KDE for a reliable optimization signal.

Table C: Alignment performance of N=12N=12 and N=1N=1

StepPickScore (N=12N=12)PickScore (N=1N=1)Aesthetic (N=12N=12)Aesthetic (N=1N=1)
0 (Base)0.210650.210655.3745.374
1000.216660.209635.7395.272
2000.219480.209165.8475.243
3000.219790.208805.8675.226

W6) Diversity of generated images

no corresponding image diversity or coverage evaluation is reported.

We evaluate diversity using average pairwise LPIPS and DreamSim distances following [1, 2], computed for all methods on the same samples used in Table 10 of the paper. FestDPO has lower diversity than DrPO but achieves greater gains in the target reward, whereas DrPO improves it only marginally. PSO improves PickScore but exhibits a larger reduction in diversity and a lower final PickScore than FestDPO. We also provide an additional qualitative result of sample diversity of FestDPO in Appendix H. These results show that FestDPO can optimize target reward while DrPO fails to, exhibiting a better reward-diversity trade-off than PSO.

Table E: Alignment performance and diversity across prompt sets (4 seeds)

Prompt setMethodPickScore ↑LPIPS ↑DreamSim ↑
Pick-a-Pic v2SDXL-Turbo (base)22.370.62680.1845
PSO22.460.61040.1680
DrPO22.380.63520.1926
FestDPO (ours)22.600.61610.1718
Parti-PromptsSDXL-Turbo (base)22.780.62430.1906
PSO22.860.61100.1761
DrPO22.780.63420.1983
FestDPO (ours)22.920.61600.1808

Q3) Necessity of encoder

In high-dimensional settings, is the use of a pretrained encoder unavoidable?

Although not theoretically unavoidable, KDE in feature space is much preferable because kernel similarities in high-dimensional pixel space may poorly reflect the underlying data manifold and require enormous samples for estimation [3]. We therefore follow prior works of distribution matching in feature space [4, 5].

[1] Kim, Sunwoo, Minkyu Kim, and Dongmin Park. "Test-time alignment of diffusion models without reward over-optimization." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Kang, Hyeongyu, et al. "Diffusion fine-tuning via reparameterized policy gradient of the soft q-function." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Theis, Lucas, Aäron van den Oord, and Matthias Bethge. "A note on the evaluation of generative models." arXiv preprint arXiv:1511.01844 (2015).

[4] Li, Chun-Liang, et al. "Mmd gan: Towards deeper understanding of moment matching network." Advances in neural information processing systems 30 (2017).

[5] Deng, Mingyang, et al. "Generative modeling via drifting." arXiv preprint arXiv:2602.04770 (2026).

Comments on the paper

1 comment

General Response

We thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the following strengths:

  1. FestDPO addresses a practical obstacle to aligning few-step generators (jA8U, QuSw).
  2. FestDPO is simple and effective (jA8U, GYuk).
  3. The paper is clearly written (jA8U, GYgk).
  4. FestDPO improves preference alignment across three domains (jA8U, QuSw, GYgk).

In response to the reviewers’ feedback, we have conducted additional experiments and revised the paper, as summarized below.

Additional Experiments

  • Single-sample ablation (N=1N=1): The single-sample logistic preference objective degrades performance during training, supporting the importance of multi-sample KDE beyond attraction and repulsion alone.
  • Wall-clock comparison: FestDPO outperforms DrPO at every evaluated time point and surpasses PSO from eight hours onward, continuing to improve as PSO plateaus.
  • Diversity analysis: We report pairwise LPIPS and DreamSim distances alongside reward scores. FestDPO has better reward-diversity trade-off than baselines.
  • Gradient reliability: Averaged gradients computed with N=12N=12 achieve a cosine similarity of 0.834 with a 2048-sample reference.
  • Adaptive bandwidth selection: We evaluate the median heuristic as a practical starting point for bandwidth selection, reducing the need for manual tuning.
  • Bandwidth sensitivity: FestDPO improves upon the base model across all tested bandwidth configurations in image alignment.

Paper Revisions

  • Clarification of feature-space objective: We explicitly identify feature-space surrogate and distinguish it from the raw-space objective analyzed in our asymptotic results.
  • Expanded the theoretical discussion: We add analyses of finite-sample estimation error and fixed-bandwidth bias.
  • Extended the 1-D toy comparison: We report that PSO and DrPO do not recover the target distribution in toy experiment despite extensive hyperparameter tuning.
  • Incorporated the additional experiments: We include the ablation, computational comparison, diversity evaluation, gradient analysis, and bandwidth experiments described above.

We thank the reviewers again for helping us clarify the scope of our claims and strengthen the experimental evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.