- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies reward-based fine-tuning of flow models using signed advantages. The authors explain that reward weighted regression suffers from distributional collapse, and does not provide a signal to suppress low-reward samples, and how signed advantages can overcome the last issue but can make the objective unbounded. They provide a unified perspective of previous methods under this formulation, and they propose Stable Signed Flow Reinforcement (SIFR) to stabilize signed advantage regression. SIFR is formulated as a constrained optimization problem, where the constraint limits the velocity change relative to a reference model and the regularization multiplier is adapted from the measured change. SIFR is tested across three model backbones, several reward settings, and evaluation settings. The experiments show a consistent improvement over DiffusionNFT and Reinforce Adjoint Matching.
Critical strengths
-
Theoretical unification and proposed improvement: Table 1 provides an insightful theoretical unification of several different reward fine-tuning methods, and can be used to develop further insights. Beyond this, the authors tackle a known problem of signed-advantage regression, and the solution is neat.
-
Broad empirical validation: They test three different model backbones, with both model-based and rule-based rewards, evaluating them on the training reward objectives, held-out rewards, and held-out prompts. The results are extensive and positive. The ablations help clarify the contributions of the methodological choices, for example multiplier adaptation through feedback control.
Critical weaknesses
-
Clarity of the contribution: The presentation of the first three sections makes it difficult to distinguish the problem setting, motivation, background, and the specific contribution. I would expect a clearer introduction to the problem and the problem setting, and the background to be separated from the contribution. The authors should refer to previous works whenever the stated properties and statements have been previously discussed and developed. Additionally I would expect a more detailed comparison with AdvantageFlow and Adaptive Divergence Regularized Policy Optimization (ADRPO), and how SIRF improves upon them. This should be accompanied by an experimental validation.
-
Experimental settings and presentation: The multi-reward settings are unclear. If rewards are combined as is, the variance of PickScore could dominate the other two rewards, potentially being very close to the single-reward setting. Additionally, I would expect the speed comparison to use GPU hours as the x-axis, as the number of gradient updates does not directly quantify the computational cost. Moreover, a fair comparison would be between methods that achieve the same level of reward, not different levels.
Questions for authors
-
Contribution: Can you better clarify throughout the manuscript which is your contribution, and separate it to the established background?
-
Comparison to additional methods: Could you clarify the relationship to ADRPO and provide empirical comparison between SIRF, AdvantageFlow, and ADRPO?
-
Multi-reward settings: Are the 3 rewards (PickScore, ClipScore, and HPSv2.1) scaled or normalized before being combined? What is the scale for each single reward at training time?
-
Computational efficiency: Could you report the computational budget required for each method to reach a specific reward level, alongside the training rounds?