- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies reward alignment for one-step deterministic generators when gradients of the generator or reward are unavailable. It proposes ZeNO, which formulates noise optimization as a path-integral control problem in the input noise space. At each step, it samples perturbations around the current noise, weights them by their reward relative to the mean, and moves the noise in that direction under an OU reference process, which the authors connect to Langevin sampling of a reward-tilted distribution. Experiments on SDXL-Turbo, DMD2, LCM and DFGAN with aesthetic, PickScore and GenEval rewards, and on a one-step protein backbone generator with a non-differentiable designability reward, show improvements over Best-of-N and competitive results with gradient-based noise optimization.
Critical strengths
The method needs only reward values and treats both the generator and the reward as black boxes. This makes it applicable where gradient-based noise optimization cannot be used. The protein experiment shows this clearly. The designability reward involves ProteinMPNN, ESMFold, and structural alignment, so it's non-differentiable. But ZeNO raises the designable fraction from 0.03 to 0.15 and from 0.14 to 0.32.
Critical weaknesses
-
Baselines missing. ZeNO is close to standard zeroth-order gradient estimation, as in evolution strategies [1] and CMA-ES [2], with an added OU noise term and norm renormalization. Zero-Order Search [3] also searches the noise space this way. The only gradient-free baseline in the paper is Best-of-N. Could the authors compare ZeNO with those baselines under the same number of generator calls? This would show how much the path-integral formulation adds beyond these simpler methods.
-
Reward hacking. In the experiments, the reward used for optimization is also the reported metric, and the reported result is the best state over 200 rounds. With 3,200 generator calls per sample, I wonder whether part of the gain may come from overfitting the reward rather than from better samples. Could the authors share their view on whether reward hacking occurs with ZeNO? It would be helpful to see results on a held-out metric that is not used during optimization or some qualitative examples from late in the trajectory.
[1] https://arxiv.org/abs/1703.03864
[2] https://arxiv.org/abs/1604.00772
[3] https://arxiv.org/abs/2501.09732
Questions for authors
-
How does ZeNO compare with Zero-Order Search, ES, and CMA-ES under the same number of generator calls?
-
Do the authors observe reward hacking with ZeNO? How do the results look on a held-out metric not used during optimization?