- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes StepKL for RL fine-tuning of text-to-image diffusion and flow-matching models. Instead of adding a KL penalty to the loss, it defines a per-denoising-step "anchoring reward" equal to the negative squared (or L1) discrepancy between the reference and trainable models' noise/velocity predictions, and adds it to the (backfilled) task reward. The anchoring reward is gated by an adaptive weight g_k that is nonzero only when the current step's deviation exceeds a trajectory-history threshold (mean + std of past deviations) and the step's quality gain, measured by scoring the predicted clean image x0_hat with a quality model Q, falls below a history threshold (mean - std of past gains). Experiments on SD3.5 and FLUX.1-dev (with MixGRPO/FlowGRPO/DiffusionNFT/FlashGRPO as baselines) and on SD1.4/1.5/2.1 (DDPO/DPOK/D3PO) report higher target rewards at matched sample budgets, better cross-reward transfer, a 12-metric table where StepKL ranks first on 11 metrics, reduced reward hacking (t-SNE, OCR case study, VLM pairwise judge), stability under high SDE noise where KL-as-loss collapses, and lower compute to reach a target GenEval score: 6.11 versus 14.91 GPU-hours for FlowGRPO in Figure 5.
Critical strengths
- The idea of turning the reference constraint into a step-level, trajectory-adaptive reward is natural for diffusion RL, where the final-reward backfilling makes credit assignment weak, and the breadth of settings (two parameterizations, five backbones, four prompt sets, several rewards, plug-in to two GRPO variants) is a plus. The high-noise experiment in Figure 9, where KL-as-loss collapses and StepKL keeps training, is the most convincing single result.
Critical weaknesses
-
Important implementation details remain unclear. Appendix B.5.1 compares GenEval and PickScore for intermediate scoring, but the scorer used in each main experiment, the anchoring coefficient, the gating warm-up length, and the reduction or normalization of prediction discrepancies are not fully specified. The paper should also explain how the shaped rewards enter timestep-wise advantages and whether the anchoring reward and gates are detached during optimization. These details are needed to reproduce the method and understand its comparison with KL-as-loss.
-
The robustness of the comparisons is not sufficiently documented. The number of independent training runs and variability across training seeds are not clearly reported. The adaptive KL-as-loss and shuffled-gate controls are useful, but a direct quantitative comparison with StepKL under documented tuning budgets and multiple seeds is still needed. Evaluation details are also missing, including the FID reference set, metric sample counts, and VLM judge configuration. These gaps make the broad superiority claims difficult to assess and are my main reason for leaning toward rejection.
Questions for authors
-
Which intermediate scoring model is used in each main experiment? Which evaluation metrics, if any, are also used by the gate during training?
-
Please provide the anchoring coefficient, gating warm-up length, discrepancy reduction/normalization, and exact advantage construction. Are the anchoring reward and gates detached during optimization?
-
Can you extend the adaptive KL-as-loss control in Figure 6 to a direct quantitative comparison with StepKL, using documented coefficient tuning and multiple training seeds?
-
Please report the training-seed counts and uncertainty for the main results and gate ablations, together with metric sample counts, the FID reference set, and the VLM judge model and evaluation protocol.
-
Section 3.6 excludes startup and evaluation time. Does the training compute in Figure 5 include all reference-model forward passes, intermediate decoding, and reward-model calls used by the adaptive gate? Please provide the timing breakdown and training/evaluation configurations.
11 comments
### W1. Implementation details and reproducibility
Reviewer comment (excerpt):
Appendix B.5.1 compares GenEval and PickScore for intermediate scoring, but the scorer used in each main experiment, the anchoring coefficient, the gating warm-up length, and the reduction or normalization of prediction discrepancies are not fully specified. The paper should also explain how the shaped rewards enter timestep-wise advantages and whether the anchoring reward and gates are detached during optimization.
We thank the reviewer for identifying these implementation details. We have completed the specification in Table 5, Appendix C, and Appendix D.3. Q1–Q3 explain each item below.
- Scorers and hyperparameters. The intermediate scorer used in each experiment is specified in Q1, point 1. The anchoring coefficient and gate warm-up length are given in Q2, point 1; discrepancy reduction and normalization are given in Q2, point 2.
- Timestep-wise advantages and gradients. The complete construction of the advantages incorporating the shaped rewards is given in Q2, point 3 and Appendix C. Detaching the anchoring reward and gates is explained in Q2, point 4.
- Reproduction and KL-as-loss comparison. The reproduction parameters are given in Q2, point 1. The distinction from KL-as-loss is explained in Q2, point 4 and Q3.
Revision location: Appendix B.1, Table 5 (hyperparameters), PDF p. 16; Appendix C, Algorithm 1, and Listing 1 (implementation), PDF pp. 24–25; Eqs. (14)–(15) (policy objective), PDF p. 25; Appendix D.3 (scoring protocol), PDF p. 30.
### W2. Multi-seed robustness and evaluation protocol
Reviewer comment (excerpt):
The adaptive KL-as-loss and shuffled-gate controls are useful, but a direct quantitative comparison with StepKL under documented tuning budgets and multiple seeds is still needed. Evaluation details are also missing, including the FID reference set, metric sample counts, and VLM judge configuration.
We thank the reviewer for this concern. We provide multi-seed results, coefficient-tuning details, and the evaluation protocol, with the correspondence below.
- Seed counts and variability. Table 10 reports main-result statistics over five seeds. Table 3 and Figure 25 report the controls over two training seeds; see Q4, point 1.
- Direct comparisons and tuning. The direct comparison between StepKL and KL-as-loss, together with coefficient tuning, is given in Q3. Seed counts for the gate ablations are reported in Q4, point 1.
- Evaluation details. Metric sample counts, the FID reference set, and the VLM judge configuration are given in Q4, points 2(1), 2(2), and 2(3), respectively.
Revision location: Section 3.7, Table 3 (two-seed comparison), PDF pp. 9–10; Appendix D.1, Tables 8–10 (tuning and seeds), PDF pp. 26–28; Figure 25 (training controls), PDF p. 27; Appendix D.3 (evaluation protocol), PDF p. 30; Table 13 (sample counts), PDF p. 31.
Our Sincere Thanks to Reviewer X2Dk for Your Time
Dear Reviewer X2Dk,
First, we sincerely appreciate the considerable time and effort you have devoted to reviewing our work and providing detailed feedback. We have made every effort to carefully address all your questions and concerns, providing corresponding clarifications and supporting evidence in our rebuttal. We are pleased to have had the opportunity to resolve the issues you raised. If you find our responses satisfactory and believe that your concerns have been adequately addressed, we would sincerely appreciate your consideration of raising your score. Should you have any further questions or concerns, please do not hesitate to let us know. We will respond promptly with detailed explanations. Thank you again for your time and thoughtful evaluation.