- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose diffusion fine-tuning technique that combines multiple previous works, such as GRPO, DPO, and Diffusion-NFT. The authors propose two main contributions, CRD which aims to minimize the explicit reward with implicit reward with ELBO (though not exactly) approximation, and CFG-based KL regularization term to penalize the model from deviating too far from the CFG distribution of the base model. The paper compares with different diffusion fine-tuning on image generation tasks, improving quality and compositionality in some cases.
Critical strengths
S1. Reward-adaptive KL anchoring. Scaling the initial-reference KL coefficient by the sample’s raw reward is a simple, practical idea. The ablation study shows promising results, though it is questionable whether this alone can solve the reward hacking problems for different models or applications. However, I assume that this can also suffer when the group size (K) is small, where the reward can be biased.
Critical weaknesses
W1. The derivations are loose, and the key approximations are not validated. The paper presents derivations from the stated objective to the final training loss relies on a series of approximations that are neither justified theoretically nor checked empirically. In particular, the paper frames the intractable normalization constant as the central bottleneck. However, for diffusion models, the real difficulty is the intractable log-likelihood itself. In fact, the vanishing normalization constant is hardly can be seen as a novel breakthrough. The paper sidesteps this by substituting a surrogate for the log likelihood with , which is not a proper ELBO, so it is unclear what quantity is actually being optimized. There seems to be a lack of analysis or experiment that quantifies how tight this surrogate or how sensitive the resulting policy is.
W2. The CFG-based KL term and the per-sample beta are inconsistent with the stated objective. Replacing the KL to the reference model with a CFG-based KL changes the regularizer and therefore no longer corresponds to the entropy-regularized reward objective or its derived optimal distribution in Eq. 1. Additionally, introducing a per-sample beta coefficient further deviates the method from that objective. These deviations should be stated explicitly or presented with more theoretical insights.
W3. The overall justification for the proposed design is weak. Taken together, the derivation reads as a post-hoc rationalization of a set of heuristics rather than a principled method. After reading the paper, I am still not convinced why, or under what conditions, diffusion RL fine-tuning should benefit from this particular combination of ingredients.
W4. The empirical gains are marginal. The quantitative results are comparable to existing baselines, which does not provide strong evidence for the effectiveness of the proposed choices and raises concerns on their justification. The paper would be more strengthened by providing more evidence why each componenet has to be designed in the proposed specific manner, and how each compoenent affects the final performances.
Questions for authors
Can the authors present more evident on why their proposed method still holds theoretically, or at least empirically plausible? Additionally, how does the proposed method behave with and without each component shown in the method section? What is more critical choice and what is the intuition behind this? There are some ablations, but seems like internal study, rather than full comparison. More extensive ablation study would clearly show the effectiveness of the method.