- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper generates multiple samples per prompt, uses reward gradients to construct candidate moves, and selects a candidate based on capped reward minus a quadratic displacement penalty. It then uses the selected displacement to construct a regression loss for training. Under a 100-update budget, the method reports better results on the training reward metrics than ReFL and DiffusionNFT.
Critical strengths
- Under the 100-update setting, the method outperforms ReFL and DiffusionNFT on the training reward metrics. The training curves also show faster improvement in the early stages.
Critical weaknesses
- Overall, the method lacks a natural derivation from first principles and includes many design choices that seem ad hoc to me. I find it difficult to understand the motivation and mathematical rationale behind these choices. The method also has too many hyperparameters that need to be set or tuned, such as the reward-cap quantile (0.75), the global cap floor, the proposal step sizes (0.1, 0.2, 0.4), the acceptance-rate target used to adjust the displacement penalty, the keep-loss weight, and the intermediate time step ().
- The paper provides limited evidence that the method achieves better final performance than ReFL or DiffusionNFT. Table 3 compares methods under a fixed 100-update budget rather than after convergence. Table 2 compares MEND with released checkpoints trained with different reward objectives and numbers of updates.
- The paper constructs candidates using reward gradients and then uses the selected displacement to define a regression loss. This extra step feels indirect and insufficiently motivated.
Questions for authors
- The regression loss is applied at a stored state near . Why is only one intermediate time step used? How does this supervision affect the full sampling trajectory, and could the choice of time step limit the final reward improvement?