- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies joint distillation and reward-based fine-tuning for image diffusion models. The authors observe an instability when switching between distillation and RL fine-tuning when done sequentially. They then define a unified objective and optimization algorithm by blending both the target and the query from on-policy distillation and diffusion NFT. The authors show that by controlling the parameters of their loss they can recover a certain version of NFT on one end and OPD on the other. They conduct experiments by fine-tuning an image diffusion model using a teacher model and a suite of tasks with reward functions, showing slightly better performance and significantly faster convergence compared to OPD or NFT alone.
Critical strengths
The method is grounded in literature and the authors have a good understanding of the OPD and NFT methods. The URF algorithm design presents a reasonable combination of the two.
Critical weaknesses
- The gains reported in Table 1 are slim, but the experiment, according to the appendix, is not averaged over multiple seeds. For score differences of this magnitude, a variance estimation over several training runs is necessary to establish significance, especially with the discrepancies between the starting scores of the methods (see below) and baseline scores fluctuations compared to the original papers.
- An unexplained discrepancy between the different methods’ starting performance exists on training curves, particularly those displayed in Figure 6 (a, b, c) and Figure 7 (a). In figures 6a, 6b, 6c and 7a, the score of NFT at the beginning of training is significantly lower than that of the other methods. Additionally, in figure 6c the starting quality of URF and OPD differ by an amount comparable to the final quality difference. These discrepancies put the validity of the experimental setup into question and need to be resolved or explained in the paper.
- The authors motivate the need for this method by observing an instability in training when switching between OPD and NFT. The study of switching between OPD and NFT, however, is limited to a few experiments with only one-time-switch setups. The main comparisons in this paper are with OPD-only and NFT-only methods. Given the motivation of the method through the issue of switching between OPD and NFT, I consider a larger study of different switching strategies an essential baseline.
Questions for authors
- Are there better ways to combine sequential (or mixed) OPD+NFT updates? What would be the outcome?
- What explains the different starting points for the different methods in Figures 6 and 7? What impact do these differences have on the final comparisons?
- How are the NFT baselines set up? The appendix contains detailed explanations for OPD and URF, but not NFT. Notably, the scores reported by the original NFT paper are substantially different in several of the reward functions, in some cases closely approaching URF.
- What are the CFG settings used for NFT, OPD and URF? How was this choice motivated? The appendix reports various CFG for URF on different datasets, the NFT CFG setting is not clear.
- How is the method to compute the specific speedup numbers motivated? Computing the time to reach a baselines peak could produce a misleading number, since URF reaches a higher final reward. This metric is very sensitive to the length of OPD training, with an extreme point on the plateau-ing curve at the end of training inflating the score.
- Figure 3 seems to suggest that setting URF’s p to 1 recovers the NFT method, however according to the derivation in Appendix A, it recovers only the positive NFT vector. I think this is an important distinction and should be displayed in Figure 3.
- Is the caption to figure 6 accurate? NFT never reaches the OPD peak.
I believe that the method is reasonable and grounded in the literature, however the methodology of the experimentation has severe issues that prevent the motivation and evaluation of URF from being effective. I am open to revising my score if the authors make significant improvements to the experimental suite, and explain and resolve the inconsistencies between the reported scores.
5 comments
Author Response (2/2)
[Q3] DiffusionNFT Reproducibility and Baseline Configuration
We thank the reviewer for raising concerns about DiffusionNFT reproducibility.
To improve reproducibility, we have added configuration details in Tables 4 and 5 of Appendix B.3, specifying the baseline initialization and task-specific training, teacher, and evaluation CFG settings. We follow the official DiffusionNFT implementation's native CFG-free setting (CFG 1.0) for the standalone baseline. All methods are evaluated using the corresponding benchmark prompts, with training costs reported in GPU-hours.
Differences from previously reported baseline scores may arise from inference randomness, initial noise sampling, and implementation-specific numerical procedures. We have clarified the evaluation configurations to make these comparisons more transparent.
Importantly, Table 1 shows that URF outperforms Multi-Task NFT across all three optimized rewards: GenEval (0.963 vs. 0.946), OCR (0.963 vs. 0.951), and PickScore (24.003 vs. 22.490).
Furthermore, the additional matched-compute, multi-seed experiments in [W1] consistently favor URF over NFT, providing complementary evidence of its effectiveness across independent training runs.
[Q4] Clarification of Training and Evaluation CFG
We have added a configuration table in Appendix B.3 that separately reports training CFG, teacher CFG, evaluation CFG, and initialization for the full-step single-reward experiments.
[Q5] Clarification of Convergence Speedup
We compute speedup based on the training time required to reach each baseline's peak reward, measured within 200 GPU-hours for single-reward and 300 GPU-hours for multi-reward training. We agree that the speedup criterion depends on training duration and reward fluctuations. Even beyond this speedup metric, URF's superior training efficiency is evident from the following results:
- Multi-reward training (Fig. 1): Faster convergence across all three optimized rewards.
- Single-reward training (Fig. 6): Faster reward improvement and higher final performance.
- Fixed-budget comparison ([W1]): Under 50 GPU-hours, URF outperforms OPD by 0.120 on GenEval and 0.052 on OCR across two additional seeds, including reward-related computational costs.
[Q6] Clarification of the DiffusionNFT Limit at p = 1
We thank the reviewer for highlighting this distinction. At , URF recovers the NFT-positive target, as DiffusionNFT itself follows only the positive-reward branch when . We have revised Figure 3 to highlight the NFT-positive velocity target and show that URF follows this target at .
[Q7] Clarification of Figure 6 Caption
We thank the reviewer for pointing this out. The caption's description of the experimental comparisons is accurate. DiffusionNFT trained without prior distillation converges substantially more slowly and does not reach OPD's peak reward within the 200-GPU-hour budget.
Additionally, we have corrected the reward ordering in the caption.
Dear Reviewer qz9Y,
Thank you again for your time and constructive comments. As the discussion period is approaching its close, we wanted to follow up and check whether our responses have addressed your concerns. If any concerns remain, we would be glad to address them within the remaining time. Thank you very much for your effort in reviewing our work.
Author Response (1/2)
We sincerely appreciate the reviewer's careful evaluation and constructive feedback.
In response, we conducted additional experiments on multi-seed robustness [W1] and alternative switching strategies [W3,Q1], clarified the baseline configurations and evaluation protocols [W2, Q2–Q4], and provided further technical clarifications [Q5–Q7]. We address each concern below and would be happy to clarify any remaining questions and make every effort to conduct additional experiments if needed.
[W1] Statistical Reliability and Multi-Seed Evaluation
We acknowledge that the results in Table 1 require further multi-seed statistical validation. To assess robustness, we conducted additional experiments using two independent training seeds (123 and 456), allocating 50 GPU-hours to each method.
URF consistently outperforms both baselines across the additional seeds, improving over OPD by 0.120 on GenEval and 0.052 on OCR on average under matched compute budgets.
We address the initial-score discrepancies and baseline-score fluctuations in [Q2–Q4] below and Appendix B.3.
[W2, Q2] Clarification of Initial Performance Differences
We thank the reviewer for pointing out the differences in initial performance.
For the DiffusionNFT comparisons in Figures 6(a–c) and 7(a), we use CFG 1.0 following the original DiffusionNFT configuration, whereas URF and OPD are evaluated with CFG 4.5. Since DiffusionNFT is designed for CFG-free optimization and achieved higher reward scores under CFG 1.0 than under stronger guidance in our experiments, we adopted this setting. We have clarified the evaluation CFG settings in the figures and Appendix B.3.
Regarding the remaining initial-score discrepancies between URF and OPD in figure 6(c), we reconfirmed that both methods share the same pretrained initialization and evaluation CFG. Nevertheless, differences may arise from implementation-specific sampling, initial noise difference and numerical procedures.
These initial-score discrepancies do not necessarily indicate an experimental error, as Figures 6(a) and 6(d) also show that URF can start below the baselines but ultimately achieve higher rewards. Together with its consistent convergence advantages across the evaluated settings, these results support URF's effectiveness beyond differences in initial evaluation scores.
[W3, Q1] Comparison with Alternative Switching Strategies
We thank the reviewer for suggesting a broader evaluation of switching strategies.
We previously evaluated the following switching and interpolation strategies:
We additionally evaluate nonlinear preference calibration (see Appendix C, Fig. 9 in the revised paper for the full training curves):
pγ(s)=sγ+(1−s)γsγ
where γ→∞ corresponds to sample-wise hard switching.
These experiments consistently favor the continuous linear calibration used by URF over the tested switching and interpolation alternatives.
From a theoretical perspective, linear preference calibration also provides stability-motivated properties. For any Lipschitz function f:[0,1]→[0,1] satisfying f(0)=0 and f(1)=1, its Lipschitz constant satisfies Lf≥1, with equality attained by f(s)=s. Thus, linear calibration minimizes the worst-case sensitivity of the preference weight to reward-normalization perturbations.
Moreover, Appendix A.1 establishes the Wasserstein contraction of URF's adaptive query:
W2(qKp,qfwd)≤(1−p)W2(q,qfwd)
Together, these properties motivate URF's continuous linear sample-wise adaptation without requiring a manually selected switching step. We would welcome suggestions for additional switching baselines and will make every effort to complete the requested experiments within the rebuttal period.