- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors present an approach (FestDPO) to apply direct preference optimization to few-step generative models where the likelihoods are implicit. They address this issue by using Kernel Density Estimation (KDE) to estimate the likelihoods using multiple samples generated by the frozen reference model and the model with the weights being fine-tuned. They argue that this is a tractable approach since few-step models are naturally very fast at generating samples. They present theoretical analysis showing that the FestDPO objective approaches that of DPO as the number of samples used for KDE grows. Finally they compare the effectiveness of FestDPO empirically to two recent baselines across text-to-image and protein generation modalities, showing that FestDPO leads to improved performance in terms of PickScore, Aesthetic score, and human evaluation.
Critical strengths
1- The central idea is simple, well-presented and motivated, and effective.
2- Experiments across multiple modalities show FestDPO seems promising (even though the improvements at times seem marginal e.g. Table 10)
3- well-written, easy to follow, good figures. Also includes ablations that show the robustness of the approach across N, and different encoders.
Critical weaknesses
1- I am unsure of the novelty/significance of the approach. Conceptually, FestDPO's final proposed objective seems closely related to that of DRPO as it also uses contrastive reference pairs to construct its fine-tuning objective. Furthermore, the idea of using KDE to estimate the score for implicit few step models appears in [1]. In my opinion, a more convincing argument regarding the novelty and significance of FestDPO is missing in the paper.
2- In the qualitative comparisons in the appendix (Figures 8 and 9) the two baselines (especially DrPO) seem nearly identical to the Base model. This raises concerns for me regarding whether the implementation/hyperparameters used for the baseline results were tuned properly.
3- There is a lack of discussion regarding sample complexity of the KDE, Especially as in high dimensions, we would expect to need many samples. The ablation in the experiments addresses this in somewhat in practice, but more thorough investigation and discussion, especially on the theory side would be helpful. Also, are the theoretical results/sample complexity affected by choice of pertained encoders? If so, how?
If these weaknesses are sufficiently addressed, I am open to raising my score.
Questions for authors
1- FestDPO’s final objective appears closely related to DRPO’s, since both build the fine-tuning objective from contrastive reference pairs. What is the conceptual difference between the two objectives, and in what settings would FestDPO be expected to succeed where DRPO fails?
2- To my understanding, using KDE to estimate the score for implicit few-step models has been proposed in [1] (as the authors of the current paper also cite them). Beyond combining this with preference optimization, is there anything new in how FestDPO uses KDE? Could the authors add a direct comparison or discussion to clarify the contribution?
3- In the qualitative comparisons (Figures 8 and 9), the baselines, especially DRPO, look nearly identical to the base model. How were the baseline implementations and hyperparameters chosen and tuned? Could the authors report a hyperparameter sweep or other evidence that the baselines actually moved away from the base model, for example significant reward difference or KL divergence relative to the base model?
4- KDE is known to need many samples in high dimensions. What are the sample complexity requirements of the KDE-based score estimate in this setting, and can the authors provide theoretical bounds or at least a clear discussion of how estimation error scales with dimension and number of samples? Furthermore, are the theoretical results and the sample complexity affected by the choice of pretrained encoder, for example through its embedding dimension or the geometry of its feature space? (the empirical ablations already somewhat address this concern empirically, I'm mainly interested in a theoretical or intuitive explanation)
6 comments
Yes, my apologies, I should have included this in the review:
[1]: Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inference (Lee et. al.)
W1 & Q2) Novelty and significance of FestDPO
Beyond combining this with preference optimization, is there anything new in how FestDPO uses KDE?
What is the conceptual difference between the two objectives, and in what settings would FestDPO be expected to succeed where DRPO fails?
Our core contribution is a sample-based DPO formulation for few-step generative models using non-parametric likelihood estimation, with an asymptotic connection to the reward-tilted distribution induced by the Bradley–Terry model. This contribution offers distinct capabilities and novelty beyond FAV and DrPO.
Firstly, FestDPO can learn directly from preference pairs without reward evaluations or gradients, while FAV requires an explicit differentiable reward. This property of FestDPO avoids fitting a separate reward model, which can require substantial data and careful tuning [2,3,4], Therefore, FestDPO is applicable when preferences are available, but a reliable reward model is difficult to obtain.
Second, DrPO does not follow the Bradley–Terry preference model and shows limited improvement in our offline experiments. The target distribution of DrPO is defined by a contrastive energy and does not generally coincide with the reward that determines the preference. Therefore, DrPO cannot proportionally sample from while FestDPO can. Empirically, in the 1-D toy setting, DrPO fails to sample in proportion to despite an extensive hyperparameter search. These results are shown in Figure 7 of the appendix. In image and protein tasks, FestDPO significantly outperforms DrPO in the offline setting, where no additional reward queries are available during training. This is particularly important in the standard DPO setting, where offline preferences are available but a reliable reward function is inaccessible.
[2] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." NeurIPS 2023: 53728-53741.
[3] An, Gaon, et al. "Direct preference-based policy optimization without reward modeling." NeurIPS 2023: 70247-70266.
[4] Zhang, Yang, and Jeffrey Skolnick. "Scoring function for automated assessment of protein structure template quality." Proteins: Structure, Function, and Bioinformatics 57.4 (2004): 702-710.
W2 & Q3) Faithfulness of baseline reproduction
How were the baseline implementations and hyperparameters chosen and tuned?
For each baseline result of the paper (PSO and DrPO), we conducted a grid search based on the hyperparameters recommended in the original paper and selected the best-performing configuration. As demonstrated in Table G, DrPO does not improve its reward in the offline setting over grid search. For PSO, we selected the indicated best-performing combination of and . We did not undertune the baselines to favor our method.
Table G: Hyperparameter tuning of DrPO on SDXL-DMD2 (4 seeds).
| PickScore | ||||
|---|---|---|---|---|
| =0.1 | 0.2117 | 0.2117 | 0.2117 | 0.2114 |
| =0.2 | 0.2113 | 0.2112 | 0.2114 | 0.2111 |
| =0.3 | 0.2109 | 0.2107 | 0.2110 | 0.2120 |
| =0.5 | 0.2107 | 0.2107 | 0.2108 | 0.2107 |
Table H: Hyperparameter tuning of PSO on SDXL-DMD2 (4 seeds).
| PickScore | ||||
|---|---|---|---|---|
| =1 | 0.2138 | 0.2107 | 0.2142 | 0.1844 |
| =10 | 0.2137 | 0.2144 | 0.2158 | 0.2087 |
| =50 | 0.2115 | 0.2120 | 0.2123 | 0.2133 |
| =100 | 0.2112 | 0.2114 | 0.2116 | 0.2123 |
W3 & Q4) Sample complexity of KDE
How estimation error scales with dimension and number of samples?
Are the theoretical results/sample complexity affected by choice of pertained encoders?
For a fixed bandwidth , both the KDE density and its gradient can be expressed as Monte Carlo sample averages. Under standard regularity conditions, both estimators exhibit the following convergence rates [5,6]:
The score estimator inherits the convergence rate of the density estimator. Although KDE in feature space may lose information from the raw space, a pretrained encoder can reduce the dimensionality of the space, mitigating the sample complexity of KDE. The choice of encoder affects the embedding dimension and feature-space geometry, but our ablation study shows that FestDPO remains robust across different pretrained encoders, including Latent-MAE, CLIP, and DINOv2.
[5] Mohamed, Shakir, et al. "Monte carlo gradient estimation in machine learning." JMLR 21.132 (2020): 1-62.
[6] Ascoli, Ruben G. "Limitations of richardson extrapolation for kernel density estimation." arXiv preprint arXiv:1812.08619 (2018).
Thank you for your thoughtful review!
Before addressing your concerns, I would like to clarify one point. Could you please specify which paper you are referring to as [1]?