- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the high rollout and reward-evaluation costs from low update-to-data ratio of diffusion and flow model alignment by proposing Value-Implicit Reward Matching (VIRM). VIRM reformulates KL-regularized optimal tilted distribution to derive implicit reward of a given sample from the current policy, then apply regression between the implicit and actual reward to induce the reward-tilted distribution. This enabled to justify re-using old off-policy samples to achieve high update-to-data ratio and data efficiency. Experiments on SD3.5 and FLUX.2 report improved compositional alignment and out-of-domain performance, especially improving GenEval2 for FLUX.2 to 65.4(baseline 55.2).
Critical strengths
- The paper presents a distinct approach to diffusion/flow model fine-tuning through implicit reward matching, apart from many RL-inspired(e.g., flowGRPO) or RWR-based(e.g., diffusionNFT) methods.
- The primary result suggests faster and more effective reward alignment against other state-of-the-art baselines.
Critical weaknesses
I'd like to revise the score if the following are resolved in the rebuttal.
- Insufficient justification of : The theoretical normalizer shall be a scalar, but the paper didn't adequately justify the design of to have the same size as the data, or its prediction through an auxiliary model head.
- Several unprovided results about the following limit the soundness of the proposed method:
- Diversity is assessed with several sample images only, without quantitative metrics such as Vendi[1] or DreamSim[2] diversity score.
- Diffusion fine-tuning methods are sensitive to many factors(e.g., #samples per prompt, #prompts per iteration, batch size, and #update steps per iteration), where these details are not reported; it is unclear whether they were matched with the baselines, and hard to reproduce the results.
[1] The Vendi Score: A Diversity Evaluation Metric for Machine Learning, TMLR 2023
[2] DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data, NeruIPS 2023
Questions for authors
- Since is a reward-specific quantity independent of the generative model, is it necessary to predict by an additional head on the generative model?
- Could the authors provide an ablation result where is learned as a model-independent scalar?
- Since methods that use a forward process(e.g., DiffusionNFT) utilize multiple training samples from a single sampled clean image , isn't it misleading to characterize such methods as having UTD = 1?