acceptodds
Withdrawn from ICLR 2027

Gradient-Free Noise Optimization for Reward Alignment in Generative Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing reward alignment methods for diffusion and flow models rely on multi-step stochastic trajectories, making them difficult to extend to deterministic generators. A natural alternative is noise-space optimization, but existing approaches require backpropagation through the generator and reward pipeline, limiting applicability to differentiable settings. To address this, here we present ZeNO (Zeroth-order Noise Optimization), a gradient-free framework that formulates noise optimization as a path-integral control problem, estimable from zeroth-order reward evaluations. When instantiated with an Ornstein–Uhlenbeck reference process and a one-step receding horizon, the repeated update connects to Langevin dynamics targeting a reward-tilted distribution. ZeNO enables effective inference-time scaling and demonstrates strong performance across diverse generators and reward functions, including a protein structure generation task where backpropagation is infeasible.

Primary area:
generative models
Keywords:
Diffusion models, reward alignment, noise optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TFTROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reward alignment for one-step deterministic generators when gradients of the generator or reward are unavailable. It proposes ZeNO, which formulates noise optimization as a path-integral control problem in the input noise space. At each step, it samples perturbations around the current noise, weights them by their reward relative to the mean, and moves the noise in that direction under an OU reference process, which the authors connect to Langevin sampling of a reward-tilted distribution. Experiments on SDXL-Turbo, DMD2, LCM and DFGAN with aesthetic, PickScore and GenEval rewards, and on a one-step protein backbone generator with a non-differentiable designability reward, show improvements over Best-of-N and competitive results with gradient-based noise optimization.

Critical strengths

The method needs only reward values and treats both the generator and the reward as black boxes. This makes it applicable where gradient-based noise optimization cannot be used. The protein experiment shows this clearly. The designability reward involves ProteinMPNN, ESMFold, and structural alignment, so it's non-differentiable. But ZeNO raises the designable fraction from 0.03 to 0.15 and from 0.14 to 0.32.

Critical weaknesses
  • Baselines missing. ZeNO is close to standard zeroth-order gradient estimation, as in evolution strategies [1] and CMA-ES [2], with an added OU noise term and norm renormalization. Zero-Order Search [3] also searches the noise space this way. The only gradient-free baseline in the paper is Best-of-N. Could the authors compare ZeNO with those baselines under the same number of generator calls? This would show how much the path-integral formulation adds beyond these simpler methods.

  • Reward hacking. In the experiments, the reward used for optimization is also the reported metric, and the reported result is the best state over 200 rounds. With 3,200 generator calls per sample, I wonder whether part of the gain may come from overfitting the reward rather than from better samples. Could the authors share their view on whether reward hacking occurs with ZeNO? It would be helpful to see results on a held-out metric that is not used during optimization or some qualitative examples from late in the trajectory.

[1] https://arxiv.org/abs/1703.03864
[2] https://arxiv.org/abs/1604.00772
[3] https://arxiv.org/abs/2501.09732

Questions for authors
  • How does ZeNO compare with Zero-Order Search, ES, and CMA-ES under the same number of generator calls?

  • Do the authors observe reward hacking with ZeNO? How do the results look on a held-out metric not used during optimization?

Reviewer_kGBNOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors proposes ZeNO, a gradient-free method that improves rewards for one-step deterministic generators by optimizing their input noise. To this end, they frame the update as path-integral control under an Ornstein–Uhlenbeck reference process, which reduces to a Langevin-like step toward a reward-tilted noise distribution estimated purely from reward evaluations.

Critical strengths
  1. The method is gradient-free, so it does not require differentiable rewards and also works for non-differentiable rewards.
  2. Even on differentiable rewards, it matches or outperforms gradient-based methods.
  3. On the theoretical side, the method relies on the Langevin formulation and thus appears theoretically well founded.
  4. The method advocates a direct (one-step) optimization of the mapping from noise samples to the final output and hence does not hinge a more fragile multi-step iterative formulation.
Critical weaknesses
  1. The paper fails to mention and compare to very relevant prior work. Most importantly, it does not cite the paper "Trust-Region Noise Search for Black-Box Alignment of Diffusion and Flow Models" (Schweiger, Ram, Cremers ECCV 2026). That paper already advocates a gradient-free approach for optimizing general (also non-differentiable) reward functions in a direct one-step approach. So most of the differentiators of this method are already covered in the work of Schweiger et al. As a result, the novelty of this paper is significantly smaller.
  2. The one-step solution is achieved by a path-integral formulation. As a consequence, when you unpack the path-integral approach, the mean over the particles is really nothing but the standard evolution strategy (Gaussian smoothing, zeroth order gradient estimation, etc). At the same time, the paper does not compare to ES, CMA-ES or other derivative-free approaches.
  3. The method is not clearly superior to existing approaches. On the GenEval position task, it seems to loose out against the Best-of-N approach.
Questions for authors

Why have you not mentioned and compared to other derivative-free approaches such as Schweiger et al. ECCV 2026? Please comment on weakness 2 and explain to what extent you believe that the path-integral approach is actually different from the standard evolution strategies.

Reviewer_sPk1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing reward alignment methods are mainly designed for multi-step stochastic trajectories and differentiable rewards. To address reward alignment for one-step generative models with non-differentiable rewards, this paper proposes ZeNO (Zeroth-order Noise Optimization), which optimizes the initial noise of one-step generative models to align generation with the reward. By formulating noise-space optimization as a path-integral control (PIC) problem, ZeNO iteratively uses reward-weighted combinations of sampled noise perturbations to improve reward alignment. Experiments on image and protein generation demonstrate its effectiveness in inference-time scaling.

Critical strengths
  1. The paper is well-motivated. The core motivation is relevant to the growing adoption of few-step generative models. Black-box reward evaluation and gradient-free optimization can be widely applied across various domains.
  2. The experiments cover multiple models and domains, demonstrating that ZeNO can be integrated with different generative models and applied to diverse tasks.
Critical weaknesses
  1. Extensive inference-time scaling for reward alignment remains a computational bottleneck for few-step generators. It is unclear how few-step generation with inference-time optimization compares with multi-step generation using lightweight reward alignment in terms of both performance and computational efficiency.
  2. The proposed method is essentially an iterative search algorithm operating in the initial noise space. Therefore, comparisons with established black-box optimization methods beyond Best-of-N are necessary. Moreover, the current experiments do not sufficiently demonstrate the advantages of formulating noise-space optimization as a path-integral control problem over alternative optimization or control methods.
Questions for authors
  1. Does the proposed method offer advantages in both performance and computational efficiency compared with reward alignment methods based on multi-step generative models?
  2. Does ZeNO outperform existing iterative black-box optimization methods under the same computational budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.