- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how to align deterministic one- or few-step generators by optimizing only the initial noise when the reward is available only as a black box. It proposes ZeNOVA, which estimates the gradient of an annealed soft value from reward-weighted Gaussian perturbations, updates the noise with separate angular and radial Langevin steps to keep it near the Gaussian shell, and adds Metropolis–Hastings jumps from the prior for global exploration. Experiments on SDXL-Turbo and DMD2 with five rewards (ImageReward, PickScore, GenEval, counting and OCR) report higher rewards than Best-of-K and zeroth-order baselines, and experiments on the rCM video model with VideoAlign report gains over the base model.
Critical strengths
The motivation and analysis are clear and interesting.
Breaking down why a naive zeroth-order Langevin approach fails into three failure modes (uninformative black-box rewards, unstable few-particle estimates that push the noise off the Gaussian shell, and poor exploration from bad initial states) gives a useful lens on the problem. Especially, Fig. 2 illustrates the second one well.
Critical weaknesses
Weakness 1. The method overlaps heavily with ZeNO, and in the regime actually used it is not the sampler it is presented as.
ZeNO (Kim et al., 2026) already derives a gradient-free noise update from path-integral control with an OU reference, relates it to Langevin dynamics on the reward-tilted distribution, and evaluates it on SDXL-Turbo and DMD2 with N = 16 and a √d norm constraint. Its update is essentially the ZO Langevin baseline used here (the same NES-type linear estimator with an OU step) followed by rescaling to norm √d. The paper cites this work only as a fixed-radius projection baseline in the toy experiment and never compares to it. None of the zeroth-order baselines in Table 1 matches it either. ZO Langevin has no norm constraint and the exponential-map baseline has no noise, so the most direct competitor, which already handles failure mode II through norm rescaling, is missing. The remaining differences are the softmax-weighted estimator, the σ schedule, the radial Langevin and MH jumping. ZeNO reports that exponential weighting performs substantially worse than its linear estimator, which is not discussed here. Moreover, with 1/λ = 1000 (100 for counting; App. B.3), the softmax weights in Eq. 13 are effectively one-hot, or uniform over ties for discrete rewards (a reward gap of 0.01 already gives a ratio of e^10). MH effectively accepts only improvements or ties, and the KL regularization that motivates the framework has little effect. The method then reduces to greedy best-of-N local search with random restarts, and the smoothing argument for discrete rewards no longer holds. The angular update is also not a small step as written. Since the λ in Eq. 13 cancels the 1/λ in Eq. 15, the drift has norm η_A ρ √(1−σ²)/σ ‖P⊥ε‖, roughly 800 to 16,000 rad over the σ schedule for one-hot weights. With η_A = 0.1, even the tangent noise alone has norm √(2η_A(d−1)) ≈ 57 rad (d = 4×64×64 for SDXL-Turbo at its native 512×512). The stationarity results therefore do not describe the implemented algorithm.
Weakness 2. The three failure modes are not shown to be resolved, and the ablation suggests that random search does most of the work.
Only failure mode II is demonstrated (Fig. 2), and no norm trajectory is shown for ZeNOVA. For mode I, the soft value smooths the reward over a Gaussian kernel whose radius is annealed, which amounts to the large-δ strategy the paper itself dismisses, and no annealed-δ NES control is given. The perturbation scale of the NES baseline is not even reported (App. B.2 leaves the placeholder "σ = grad sigma"). Apart from OCR (0.376 to 0.441), the ablation gain of value annealing is small: ImageReward rises only slightly (1.698 to 1.726), PickScore even drops (23.296 to 23.187), and counting, the reward discretized for this purpose, is not ablated. For mode III, no evidence of mode switching, acceptance rates or reduced sensitivity to initialization is given. The hyperspherical update mainly prevents the collapse of the Cartesian baseline, which simple norm fixing already achieves. The ZO exponential-map baseline reaches most of its ImageReward gain (1.645 vs. 1.698) and is better on OCR (0.424 vs. 0.376), where the hyperspherical update alone even falls below ZO Langevin (0.376 vs. 0.382). In practice its effect is a separate small radial step, which preconditioning or rescaling would also provide. On OCR, the only black-box reward in Table 3, performance stays below Best-of-K (0.478) until MH jumping is added (0.441 to 0.511), and at this λ that step is random search with greedy acceptance. Since the ablation is cumulative, leave-one-out results and an MH-plus-norm-fixing control are needed.
Weakness 3. The evaluation does not show an advantage over simple search, does not rule out reward hacking, and uses outdated generators.
Best-of-K beats all three zeroth-order baselines in 9 of 10 model–reward pairs, so the headline gains over “the best zeroth-order baseline” are measured against baselines weaker than random search. Against Best-of-K, whose budget is not stated, the GenEval gain is only +0.010 and +0.005. Fig. 6 omits Best-of-K, and no wall-clock time or memory is reported anywhere, even though Best-of-K is fully parallel while ZeNOVA runs K sequential rounds. The “efficient” claim is therefore unsupported. First-order baselines run 400 to 800 iterations (ReNO’s default is 50) on different hardware, which weakens the claim that they hack the reward. Task performance on black-box rewards is measured only by the reward being optimized (besides the aesthetic score), nothing checks whether the rest of the prompt is preserved, and diversity is reported only for PickScore and without Best-of-K. At the 1/λ = 1000 setting used in Table 1, ZeNOVA has the lowest Vendi score in Fig. 7. The video results have no baseline beyond the base model, and Fig. 13 shows clear prompt drift. Finally, all image results use SDXL-Turbo and DMD2 (whose backbone is not stated, presumably SDXL). These models barely render text (base OCR 0.14 and 0.09), which inflates the gains. The paper is motivated by recent few-step and flow-map models, yet evaluates none of the modern ones such as FLUX.1-schnell or SD3.5-Large-Turbo, and a gradient-free method could even run on FLUX.1-dev, as in Ma et al. (2025).
References
- Kim et al., Gradient-Free Noise Optimization for Reward Alignment in Generative Models, arXiv:2605.11347, 2026.
- Ma et al., Scaling Inference Time Compute for Diffusion Models, CVPR 2025.
- Eyring et al., ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization, NeurIPS 2024.
Questions for authors
Please see weaknesses.
8 comments
Response to Reviewer nPyD (2)
- no -annealed NES control is given.
We provide the performance of NES with annealed in the table below. However, note that NES in Eq. (54) simply performs gradient ascent using the ZO reward gradient in Eq. (7), without accounting for the prior distribution. Using large simply has an effect for a bigger step size, where we couldn’t find a meaningful improvement.
| ImageReward | Pickscore | GenEval | Counting | OCR | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Reward | Aesthetic | Val. | Reward | Aesthetic | Val. | Reward | Aesthetic | Val. | Reward | Aesthetic | Reward | Aesthetic | Val. | |
| -annealed NES | 1.4060 | 6.0690 | 23.0238 | 22.5771 | 5.5720 | 0.8150 | 0.6592 | 5.4005 | 22.9236 | 7.475 | 5.4346 | 0.3042 | 5.4279 | 21.6683 |
| MH jumping only + norm fix | 1.6911 | 6.1210 | 23.3070 | 22.9308 | 5.5771 | 0.8733 | 0.7213 | 5.3608 | 22.9027 | 6.375 | 5.7051 | 0.4533 | 5.3873 | 21.8289 |
| ZeNOVA | 1.8078 | 6.1279 | 23.3584 | 23.4564 | 5.5957 | 1.0329 | 0.7332 | 5.4278 | 23.0361 | 0.55 | 5.7222 | 0.5114 | 5.3864 | 21.9231 |
- The perturbation scale of the NES baseline is not reported.
All zeroth-order baselines, including ZO Langevin, ZO Exponential Map, and NES, used .
- For mode III, no evidence of mode switching, acceptance rates or reduced sensitivity to initialization is given.
In all qualitative results, we compare samples generated by different methods starting from the same initial noise. While all samples generated by ZeNOVA are completely different from the original samples, ZO Langevin, ZO Exponential Map, and NES frequently produce outputs that remain perceptually similar to the originals. This demonstrates that ZeNOVA is not constrained to the region surrounding the initial noise.
- The hyperspherical update mainly prevents the collapse of the Cartesian baseline, which simple norm fixing already achieves.
As argued in the main text and illustrated by the 2D toy example, fixing the norm to a value such as or to its initial value imposes an overly restrictive constraint that renders substantial portions of the reward-tilted distribution, including high-reward regions, inaccessible. Consistent with this observation, all methods employing such constraints exhibited inferior performance compared to ZeNOVA.
- Apart from OCR (0.376 to 0.441), the ablation gain of value annealing is small: ImageReward rises only slightly (1.698 to 1.726), PickScore even drops (23.296 to 23.187).
We have already explained this phenomenon in the main manuscript; the reward landscapes of ImageReward and PickScore are continuous with many high-reward modes, allowing sufficiently high rewards to be achieved using only the local reward gradient around the initial point.
- The ZO exponential-map baseline reaches most of its ImageReward gain (1.645 vs. 1.698) and is better on OCR (0.424 vs. 0.376), where the hyperspherical update alone even falls below ZO Langevin (0.376 vs. 0.382).
We insist that cne should not expect the three components of ZeNOVA to benefit equally across all scenarios: Since the OCR reward is piecewise constant, the zeroth-order gradient estimate hardly carries any information, thus the hyperspherical update alone is insufficient. Although ZO Exponential Map achieves a score of 0.424 on SDXL-Turbo, its performance drops substantially to 0.235 on SDXL-DMD2. Moreover, all comparisons mentioned by the reviewer involve a variant of ZeNOVA without MH jumping and value annealing, which doesn’t undermine the success of the full ZeNOVA. All three components of ZeNOVA are essential, and removing one of them may still yield satisfactory performance on certain tasks but compromises the method's applicability to other black-box reward scenarios.
- On OCR, the only black-box reward in Table 3, performance stays below Best-of-K (0.478) until MH jumping is added (0.441 to 0.511).
Not only does ZeNOVA w/o MH-jumping fail to outperform Best-of-K, but all other baselines also fail. This is precisely why we introduced MH-jumping. But as we stated in Line 276, MH-jumping alone cannot outperform Best-of-K, and its effectiveness is only maximized when combined with local updates as a component of ZeNOVA.
- Leave-one-out results and an MH-plus-norm-fixing control are needed.
We provide the leave-one-out ablation results in the table below, and the performance of MH jumping only+norm fix in the table above.
| Variants | ImageReward | PickScore | OCR |
|---|---|---|---|
| ZeNOVA | 1.8078 | 23.4564 | 0.5114 |
| -MH jumping | 1.7262 | 23.1867 | 0.4412 |
| -Value annealing | 1.7945 | 23.2665 | 0.4560 |
| -Hypherspherical update | 1.7925 | 23.3534 | 0.4738 |
[continues in the following responses]
Response to Reviewer nPyD (3)
- Best-of-K beats all three zeroth-order baselines in 9 of 10 model–reward pairs, so the headline gains over “the best zeroth-order baseline” are measured against baselines weaker than random search. Against Best-of-K, whose budget is not stated, the GenEval gain is only +0.010 and +0.005.
Indeed, none of the existing zeroth-order baselines outperform Best-of-K. However, unless the reviewer can suggest a stronger alternative, it is unfair to regard this as a limitation of ZeNOVA(which consistently outperforms Best-of-K). For rewards that provide no information about the reward gradient and offer no guarantee of smoothness in the initial noise space, Best-of-K actually becomes a challenging baseline for zeroth-order methods. Since GenEval is a metric with the smallest range between 0 and 1, and considering ZeNOVA achieves the best performance across all models and tasks, we don’t think that a smaller margin on a single task constitutes a significant limitation. We also clarify that Best-of-K used the same sampling budget as all other zeroth-/first-order methods (400 or 800 samples per prompt, as stated in Line 969).
- Fig. 6 omits Best-of-K, and diversity is reported only for PickScore and without Best-of-K.
Figure 6 and 7 examines how the performance of iterative zeroth-order methods varies with the number of particles per iteration; therefore Best-of-K cannot be included in this comparison.
- No wall-clock time or memory is reported anywhere, even though Best-of-K is fully parallel while ZeNOVA runs K sequential rounds. The “efficient” claim is therefore unsupported.
Please refer to the table above in the responses on 6rQy and p26h that shows a similar wall-clock time and VRAM usage. And throughout this paper, the term “efficient” refers to sample efficiency, therefore characterizes ZeNOVA which achieves the highest performance under a fixed sampling budget. Meanwhile, although Best-of-K might be more “time-efficient” when one generates all samples in parallel, it require orders of magnitudes more VRAM than the other baselines, making it less “GPU-efficient”.
- First-order baselines run 400 to 800 iterations on different hardware, which weakens the claim that they hack the reward.
Differences in hardware are irrelevant to whether reward hacking occurs. Furthermore, we have already demonstrated the reward hacking issue in first-order methods in Table 1 of the manuscript, where improvements in the optimized reward do not translate into improvements in other validation metrics that assess the same aspect of image quality.
- Task performance on black-box rewards is measured only by the reward being optimized (besides the aesthetic score), nothing checks whether the rest of the prompt is preserved.
For the Counting task, each prompt is only "number + noun", so achieving a high reward leaves no more prompt requirements to satisfy. But since OCR prompts contain additional instructions beyond the target text, we report PickScore results for the OCR task in Table 1 of the updated manuscript. ZeNOVA achieves the highest human preference score, which also accounts for prompt alignment.
- At the 1/λ = 1000 setting used in Table 1, ZeNOVA has the lowest Vendi score in Fig. 7.
We insist that high diversity alone without high rewards cannot be considered desirable(otherwise, the base model would trivially achieve the best diversity). Instead, the reward–diversity trade-off should be evaluated through a Pareto curve, where ZeNOVA locates the top-right corner in Figure 7.
- The video results have no baseline beyond the base model, and Fig. 13 shows clear prompt drift.
Following the reviewer’s concern, we generally updated the results with more computational budget and report the comparison with Best-of-K and the strongest zeroth-order baseline for the video generation task in Table 2 of the updated manuscript.
- All image results use SDXL-Turbo and SDXL-DMD2, yet evaluates none of the modern ones.
We believe that improving a poorly performing model to achieve substantial reward gain is at least as challenging as further improving a model that already performs well. That said, we additionally report the performance of ZeNOVA on Sana-Sprint(2025)[2], a more recent few-step distilled flow-matching model with a DiT architecture, in Table 1 of the updated manuscript.
[1] Gradient-Free Noise Optimization for Reward Alignment in Generative Models, arXiv 2026.
[2] SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillation, ICCV 2025
Response to Reviewer nPyD
We greatly appreciate Reviewer
nPyDfor your detailed comments and valuable suggestions. Below, we provide thorough point-by-point responses to address your concerns.As the reviewer pointed out, the implementation of [1] reveals that it does not actually use the value gradient, but instead reduces to ZO Langevin dynamics with reward gradient and an additional norm projection to d. We didn’t include this baseline as we considered this regularization overly heuristic that produces over-constrained undesirable output distributions, as illustrated by our 2D toy experiment. We further emphasize that [1] doesn’t explain how using the value gradient alters the sampling distribution and the benefits it provides, as we establish in Proposition 1.
Nevertheless, we now provide the experimental results of ZeNO in Table 1 of the updated manuscript, where ZeNOVA still outperforms the baseline. We note that [1] originally used 3200 samples per prompt, where in our experiment it’s much less(400 or 800). Regarding the claim in ZeNO that the exponential value gradient is less effective, the authors do not specify the actual implementation of the exponential value gradient estimator, making a direct comparison difficult. We believe that their observation may be specific to their implementation, while our experiments demonstrate that the exponential value gradient achieves better performance.
While we understand the concern about the sharp reward distribution, we’d like to justify our choice with two observations: First, since the particles are generated by adding small perturbations to the current z, most particles are similar and exhibit only small reward differences. Moreover, discrete rewards such as OCR and Counting frequently assign identical scores to multiple particles. This makes a large 1/λ provide a stronger gradient signal. Second, as shown in Figure 7, ZeNOVA remains effective even with lower temperatures as low as 10. We contained the results with 1/λ=1000 in Table 1 of the manuscript by simply reporting the best performance.
As stated in Lines 255 and 991, we only propose to use a small radial update to enhance the stability towards a strict distribution of χd, not for the angular update. In fact, since the Gaussian prior is angularly symmetric, the angular update can accommodate a relatively large step size without concern of drifting out of the Gaussian prior.
We provide the evolution of the norm during ZeNOVA optimization in Figure 8 of the updated manuscript; as claimed in the main manuscript, the norm is not fixed but varies throughout optimization, yet without exploding. Failure modes I and III are demonstrated via the main experimental results, which we discuss again in the responses below.
This confusion comes from a difference between the reward gradient and the value gradient; we objected to using a large δ in the naive ZO reward gradient estimator in Eq. (6) because it is not an exact “reward gradient” for finite δ, and only converges to the true reward gradient as δ→0. In contrast, our Eq. (12) is a “value gradient estimator under an OU process”, which remains exact and unbiased for any σ∈(0,1), even when σ is substantially larger than 0. This distinction is also evident from the formulations themselves; Eq. (6) adds Gaussian noise directly to z without rescaling, causing the perturbed samples to deviate from the Gaussian prior. In contrast, Eq. (12) introduces noise through an OU process, which preserves the Gaussian prior.
In conclusion, increasing δ makes Eq. (6) less accurate, whereas Eq. (12) remains exact even for large σ.
[continues in the following responses]