- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a single-step RL fine-tuning method. The main contributions are introduction of mixture target distribution to balance off-policy and on-policy objective function, and Sinkhorn fixed-point iteration update rule, explored in previous one-step generative modeling for RL fine-tuning. The experiment results, though not comprehensive, are reasonable, and some of the proposed components seem to bring additional improvements.
Critical strengths
S1. The use of mixture target distribution is a simple, but effective practice to balance off/on-policy training in RL fine-tuning. Although it does introduce a new hyperparameter which would need further tuning for each application, model, reward, etc.
S2. The problem is timely and practical issue in industry. It can bring significant impacts to the generative modeling community for solving single-step generation and downstread reward alignment.
Critical weaknesses
W1. Novelty. Sinkhorn divergence as the objective of a Wasserstein gradient flow has already been used in prior work [1, 2]. The paper applies it to reward alignment, but its use for generative modeling itself is not new. The authors should position the contribution more precisely with respect to and clarify what is novel beyond the application setting.
W2. The theoretical results may be misleading. The Sinkhorn divergence does not guarantee convergence to the target distribution. As a result, the propositions in the paper do not characterize the method that is actually implemented.
Additionally, the proposed method does not give the same terminal distribution as the original optimal distribution. Then is it fair to compare with the baselines that fit towards the reward-tilted distribution that is different to the proposed method? This connects to W5. The mixed fixed point is a different distribution from a tempered reference tilt in theory. How should we interpret comparing these results?
Additionally, even if empirical performances are better, can we obtain a similar result from other techniques? This connects to W3.
W3. Missing comparison with other methods that mix on-policy and off-policy signals. Several recent RL fine-tuning methods, such as Diffusion-NFT [3] and dual forward KL [4], start from different objectives and derivations but arrive at a similar conclusion: combining samples from the current policy with off-policy samples. Although these methods are not designed for one-step generators, comparing against them, or at least adapting their mixing scheme to the same scheme, would clarify where the proposed method stand in the literature.
W4. Runtime comparisons. While joint reward alignment and distillation can be useful, how does this approach compare with reward alignment then distillation or the other way around? The proposed algorithm seems costly, considering drawing multiple particles from two models, and encoding them to solve Sinkhorn in the latent space. Can the authors show the performances and compute required for each approach and when one can be preferred the other?
W5. The mixture target controlled by . The mixture target is effective, as shown in Tab. 6(b), but comparable performance appears achievable by alternatively tuning the reward temperature . This does not invalidate the method, but it raises a practical question: why should a practitioner adopt the mixture formulation instead of tuning directly? Also as mentioned previously, can these be seen as fair comparison while their terminal distributions are different?
[1] Han et al., One-Step Generative Modeling via Wasserstein Gradient Flows
[2] He et al., Sinkhorn-Drifting Generative Models
[3] Zheng et al., DiffusionNFT: Online Diffusion Reinforcement with Forward Process
[4] He et al., Unifying Stable Optimization and Reference Regularization in RLHF
Questions for authors
None. Please see the weaknesses.
Dear Reviewer HiAD,
We thank you again for your detailed review. As the rebuttal period only has a few days remaining, we wanted to kindly ask if your concerns have been addressed. We have provided several new results that demonstrate
(1) RWTD's capability retention after GenEval training,
(2) RWTD's improvements relative to tuning β for the reference-only variants, and
(3) RWTD's gains on held-out metrics (i.e. CLIP, PickScore, GenEval) compared to DRaFT and FAV at matched HPSv2 optimization.
We also have appropriately toned down our interpretation of the partial transport step parameter η given your insightful comment in how it rescales the gradient.
We believe these new results and clarifications have addressed the weaknesses and questions that you brought up. If anything remains unclear, we'd be happy to provide further clarification. Thank you for your time!