acceptodds
Under review as a conference paper at ICLR 2027

Manifold-Constrained Initial Noise Optimization for Efficient Generative Model Alignment

15 views

Abstract

Recent advances in distillation and flow-map models have enabled deterministic one- or few-step generation for high-quality data, facilitating a new branch of reward alignment approaches that directly optimize the initial noise from a Gaussian distribution. However, most existing initial-noise optimization methods rely on first-order gradient information, which is either inapplicable or suffers from instability and inefficiency in black-box reward scenarios. Here, we introduce ZeNOVA (Zeroth-order Noise Optimization with Value Annealing), a stable and efficient initial noise alignment method in a gradient-free manner. Specifically, we address existing algorithms’ major challenge in black-box scenarios through annealed soft-value guidance, manifold-constrained hyperspherical Langevin dynamics, and Metropolis-Hastings jumping. Extensive experiments on image and video generative models show that ZeNOVA outperforms all evaluated zeroth-order baselines by optimizing the initial noise toward higher rewards substantially more stably while exploiting the geometry of the Gaussian prior, demonstrating its practical applicability to various black-box reward alignment.

Primary area:
generative models
Keywords:
Generative Models. Reward Alignment. Inference-time Scaling.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nPyDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to align deterministic one- or few-step generators by optimizing only the initial noise when the reward is available only as a black box. It proposes ZeNOVA, which estimates the gradient of an annealed soft value from reward-weighted Gaussian perturbations, updates the noise with separate angular and radial Langevin steps to keep it near the Gaussian shell, and adds Metropolis–Hastings jumps from the prior for global exploration. Experiments on SDXL-Turbo and DMD2 with five rewards (ImageReward, PickScore, GenEval, counting and OCR) report higher rewards than Best-of-K and zeroth-order baselines, and experiments on the rCM video model with VideoAlign report gains over the base model.

Critical strengths

The motivation and analysis are clear and interesting.

Breaking down why a naive zeroth-order Langevin approach fails into three failure modes (uninformative black-box rewards, unstable few-particle estimates that push the noise off the Gaussian shell, and poor exploration from bad initial states) gives a useful lens on the problem. Especially, Fig. 2 illustrates the second one well.

Critical weaknesses

Weakness 1. The method overlaps heavily with ZeNO, and in the regime actually used it is not the sampler it is presented as.

ZeNO (Kim et al., 2026) already derives a gradient-free noise update from path-integral control with an OU reference, relates it to Langevin dynamics on the reward-tilted distribution, and evaluates it on SDXL-Turbo and DMD2 with N = 16 and a √d norm constraint. Its update is essentially the ZO Langevin baseline used here (the same NES-type linear estimator with an OU step) followed by rescaling to norm √d. The paper cites this work only as a fixed-radius projection baseline in the toy experiment and never compares to it. None of the zeroth-order baselines in Table 1 matches it either. ZO Langevin has no norm constraint and the exponential-map baseline has no noise, so the most direct competitor, which already handles failure mode II through norm rescaling, is missing. The remaining differences are the softmax-weighted estimator, the σ schedule, the radial Langevin and MH jumping. ZeNO reports that exponential weighting performs substantially worse than its linear estimator, which is not discussed here. Moreover, with 1/λ = 1000 (100 for counting; App. B.3), the softmax weights in Eq. 13 are effectively one-hot, or uniform over ties for discrete rewards (a reward gap of 0.01 already gives a ratio of e^10). MH effectively accepts only improvements or ties, and the KL regularization that motivates the framework has little effect. The method then reduces to greedy best-of-N local search with random restarts, and the smoothing argument for discrete rewards no longer holds. The angular update is also not a small step as written. Since the λ in Eq. 13 cancels the 1/λ in Eq. 15, the drift has norm η_A ρ √(1−σ²)/σ ‖P⊥ε‖, roughly 800 to 16,000 rad over the σ schedule for one-hot weights. With η_A = 0.1, even the tangent noise alone has norm √(2η_A(d−1)) ≈ 57 rad (d = 4×64×64 for SDXL-Turbo at its native 512×512). The stationarity results therefore do not describe the implemented algorithm.

Weakness 2. The three failure modes are not shown to be resolved, and the ablation suggests that random search does most of the work.

Only failure mode II is demonstrated (Fig. 2), and no norm trajectory is shown for ZeNOVA. For mode I, the soft value smooths the reward over a Gaussian kernel whose radius is annealed, which amounts to the large-δ strategy the paper itself dismisses, and no annealed-δ NES control is given. The perturbation scale of the NES baseline is not even reported (App. B.2 leaves the placeholder "σ = grad sigma"). Apart from OCR (0.376 to 0.441), the ablation gain of value annealing is small: ImageReward rises only slightly (1.698 to 1.726), PickScore even drops (23.296 to 23.187), and counting, the reward discretized for this purpose, is not ablated. For mode III, no evidence of mode switching, acceptance rates or reduced sensitivity to initialization is given. The hyperspherical update mainly prevents the collapse of the Cartesian baseline, which simple norm fixing already achieves. The ZO exponential-map baseline reaches most of its ImageReward gain (1.645 vs. 1.698) and is better on OCR (0.424 vs. 0.376), where the hyperspherical update alone even falls below ZO Langevin (0.376 vs. 0.382). In practice its effect is a separate small radial step, which preconditioning or rescaling would also provide. On OCR, the only black-box reward in Table 3, performance stays below Best-of-K (0.478) until MH jumping is added (0.441 to 0.511), and at this λ that step is random search with greedy acceptance. Since the ablation is cumulative, leave-one-out results and an MH-plus-norm-fixing control are needed.

Weakness 3. The evaluation does not show an advantage over simple search, does not rule out reward hacking, and uses outdated generators.

Best-of-K beats all three zeroth-order baselines in 9 of 10 model–reward pairs, so the headline gains over “the best zeroth-order baseline” are measured against baselines weaker than random search. Against Best-of-K, whose budget is not stated, the GenEval gain is only +0.010 and +0.005. Fig. 6 omits Best-of-K, and no wall-clock time or memory is reported anywhere, even though Best-of-K is fully parallel while ZeNOVA runs K sequential rounds. The “efficient” claim is therefore unsupported. First-order baselines run 400 to 800 iterations (ReNO’s default is 50) on different hardware, which weakens the claim that they hack the reward. Task performance on black-box rewards is measured only by the reward being optimized (besides the aesthetic score), nothing checks whether the rest of the prompt is preserved, and diversity is reported only for PickScore and without Best-of-K. At the 1/λ = 1000 setting used in Table 1, ZeNOVA has the lowest Vendi score in Fig. 7. The video results have no baseline beyond the base model, and Fig. 13 shows clear prompt drift. Finally, all image results use SDXL-Turbo and DMD2 (whose backbone is not stated, presumably SDXL). These models barely render text (base OCR 0.14 and 0.09), which inflates the gains. The paper is motivated by recent few-step and flow-map models, yet evaluates none of the modern ones such as FLUX.1-schnell or SD3.5-Large-Turbo, and a gradient-free method could even run on FLUX.1-dev, as in Ma et al. (2025).

References

  • Kim et al., Gradient-Free Noise Optimization for Reward Alignment in Generative Models, arXiv:2605.11347, 2026.
  • Ma et al., Scaling Inference Time Compute for Diffusion Models, CVPR 2025.
  • Eyring et al., ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization, NeurIPS 2024.
Questions for authors

Please see weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer nPyD

We greatly appreciate Reviewer nPyD for your detailed comments and valuable suggestions. Below, we provide thorough point-by-point responses to address your concerns.

  • ZeNO [1] already derives a gradient-free noise update from path-integral control with an OU reference.

As the reviewer pointed out, the implementation of [1] reveals that it does not actually use the value gradient, but instead reduces to ZO Langevin dynamics with reward gradient and an additional norm projection to d\sqrt{d}. We didn’t include this baseline as we considered this regularization overly heuristic that produces over-constrained undesirable output distributions, as illustrated by our 2D toy experiment. We further emphasize that [1] doesn’t explain how using the value gradient alters the sampling distribution and the benefits it provides, as we establish in Proposition 1.

Nevertheless, we now provide the experimental results of ZeNO in Table 1 of the updated manuscript, where ZeNOVA still outperforms the baseline. We note that [1] originally used 3200 samples per prompt, where in our experiment it’s much less(400 or 800). Regarding the claim in ZeNO that the exponential value gradient is less effective, the authors do not specify the actual implementation of the exponential value gradient estimator, making a direct comparison difficult. We believe that their observation may be specific to their implementation, while our experiments demonstrate that the exponential value gradient achieves better performance.

  • With 1/λ = 1000 (100 for counting; App. B.3), the softmax weights in Eq. 13 are effectively one-hot.

While we understand the concern about the sharp reward distribution, we’d like to justify our choice with two observations: First, since the particles are generated by adding small perturbations to the current zz, most particles are similar and exhibit only small reward differences. Moreover, discrete rewards such as OCR and Counting frequently assign identical scores to multiple particles. This makes a large 1/λ provide a stronger gradient signal. Second, as shown in Figure 7, ZeNOVA remains effective even with lower temperatures as low as 10. We contained the results with 1/λ=1000 in Table 1 of the manuscript by simply reporting the best performance.

  • The angular update is also not a small step as written.

As stated in Lines 255 and 991, we only propose to use a small radial update to enhance the stability towards a strict distribution of χd\chi_{d}, not for the angular update. In fact, since the Gaussian prior is angularly symmetric, the angular update can accommodate a relatively large step size without concern of drifting out of the Gaussian prior.

  • Only failure mode II is demonstrated (Fig. 2), and no norm trajectory is shown for ZeNOVA.

We provide the evolution of the norm during ZeNOVA optimization in Figure 8 of the updated manuscript; as claimed in the main manuscript, the norm is not fixed but varies throughout optimization, yet without exploding. Failure modes I and III are demonstrated via the main experimental results, which we discuss again in the responses below.

  • For mode I, the soft value smooths the reward over a Gaussian kernel whose radius is annealed, which amounts to the large-δ strategy the paper itself dismisses.

This confusion comes from a difference between the reward gradient and the value gradient; we objected to using a large δ\delta in the naive ZO reward gradient estimator in Eq. (6) because it is not an exact “reward gradient” for finite δ\delta, and only converges to the true reward gradient as δ→0\delta\to0. In contrast, our Eq. (12) is a “value gradient estimator under an OU process”, which remains exact and unbiased for any σ∈(0,1)\sigma \in (0,1), even when σ\sigma is substantially larger than 0. This distinction is also evident from the formulations themselves; Eq. (6) adds Gaussian noise directly to zz without rescaling, causing the perturbed samples to deviate from the Gaussian prior. In contrast, Eq. (12) introduces noise through an OU process, which preserves the Gaussian prior.

In conclusion, increasing δ\delta makes Eq. (6) less accurate, whereas Eq. (12) remains exact even for large σ\sigma.

[continues in the following responses]

AuthorsOpenReview ↗

Response to Reviewer nPyD (2)

  • no σ\sigma-annealed NES control is given.

We provide the performance of NES with annealed σ\sigma in the table below. However, note that NES in Eq. (54) simply performs gradient ascent using the ZO reward gradient in Eq. (7), without accounting for the prior distribution. Using large σ\sigma simply has an effect for a bigger step size, where we couldn’t find a meaningful improvement.

ImageRewardPickscoreGenEvalCountingOCR
RewardAestheticVal.RewardAestheticVal.RewardAestheticVal.RewardAestheticRewardAestheticVal.
σ\sigma-annealed NES1.40606.069023.023822.57715.57200.81500.65925.400522.92367.4755.43460.30425.427921.6683
MH jumping only + norm fix1.69116.121023.307022.93085.57710.87330.72135.360822.90276.3755.70510.45335.387321.8289
ZeNOVA1.80786.127923.358423.45645.59571.03290.73325.427823.03610.555.72220.51145.386421.9231
  • The perturbation scale of the NES baseline is not reported.

All zeroth-order baselines, including ZO Langevin, ZO Exponential Map, and NES, used σ=0.1\sigma=0.1.

  • For mode III, no evidence of mode switching, acceptance rates or reduced sensitivity to initialization is given.

In all qualitative results, we compare samples generated by different methods starting from the same initial noise. While all samples generated by ZeNOVA are completely different from the original samples, ZO Langevin, ZO Exponential Map, and NES frequently produce outputs that remain perceptually similar to the originals. This demonstrates that ZeNOVA is not constrained to the region surrounding the initial noise.

  • The hyperspherical update mainly prevents the collapse of the Cartesian baseline, which simple norm fixing already achieves.

As argued in the main text and illustrated by the 2D toy example, fixing the norm to a value such as d\sqrt{d} or to its initial value imposes an overly restrictive constraint that renders substantial portions of the reward-tilted distribution, including high-reward regions, inaccessible. Consistent with this observation, all methods employing such constraints exhibited inferior performance compared to ZeNOVA.

  • Apart from OCR (0.376 to 0.441), the ablation gain of value annealing is small: ImageReward rises only slightly (1.698 to 1.726), PickScore even drops (23.296 to 23.187).

We have already explained this phenomenon in the main manuscript; the reward landscapes of ImageReward and PickScore are continuous with many high-reward modes, allowing sufficiently high rewards to be achieved using only the local reward gradient around the initial point.

  • The ZO exponential-map baseline reaches most of its ImageReward gain (1.645 vs. 1.698) and is better on OCR (0.424 vs. 0.376), where the hyperspherical update alone even falls below ZO Langevin (0.376 vs. 0.382).

We insist that cne should not expect the three components of ZeNOVA to benefit equally across all scenarios: Since the OCR reward is piecewise constant, the zeroth-order gradient estimate hardly carries any information, thus the hyperspherical update alone is insufficient. Although ZO Exponential Map achieves a score of 0.424 on SDXL-Turbo, its performance drops substantially to 0.235 on SDXL-DMD2. Moreover, all comparisons mentioned by the reviewer involve a variant of ZeNOVA without MH jumping and value annealing, which doesn’t undermine the success of the full ZeNOVA. All three components of ZeNOVA are essential, and removing one of them may still yield satisfactory performance on certain tasks but compromises the method's applicability to other black-box reward scenarios.

  • On OCR, the only black-box reward in Table 3, performance stays below Best-of-K (0.478) until MH jumping is added (0.441 to 0.511).

Not only does ZeNOVA w/o MH-jumping fail to outperform Best-of-K, but all other baselines also fail. This is precisely why we introduced MH-jumping. But as we stated in Line 276, MH-jumping alone cannot outperform Best-of-K, and its effectiveness is only maximized when combined with local updates as a component of ZeNOVA.

  • Leave-one-out results and an MH-plus-norm-fixing control are needed.

We provide the leave-one-out ablation results in the table below, and the performance of MH jumping only+norm fix in the table above.

VariantsImageRewardPickScoreOCR
ZeNOVA1.807823.45640.5114
-MH jumping1.726223.18670.4412
-Value annealing1.794523.26650.4560
-Hypherspherical update1.792523.35340.4738

[continues in the following responses]

AuthorsOpenReview ↗

Response to Reviewer nPyD (3)

  • Best-of-K beats all three zeroth-order baselines in 9 of 10 model–reward pairs, so the headline gains over “the best zeroth-order baseline” are measured against baselines weaker than random search. Against Best-of-K, whose budget is not stated, the GenEval gain is only +0.010 and +0.005.

Indeed, none of the existing zeroth-order baselines outperform Best-of-K. However, unless the reviewer can suggest a stronger alternative, it is unfair to regard this as a limitation of ZeNOVA(which consistently outperforms Best-of-K). For rewards that provide no information about the reward gradient and offer no guarantee of smoothness in the initial noise space, Best-of-K actually becomes a challenging baseline for zeroth-order methods. Since GenEval is a metric with the smallest range between 0 and 1, and considering ZeNOVA achieves the best performance across all models and tasks, we don’t think that a smaller margin on a single task constitutes a significant limitation. We also clarify that Best-of-K used the same sampling budget as all other zeroth-/first-order methods (400 or 800 samples per prompt, as stated in Line 969).

  • Fig. 6 omits Best-of-K, and diversity is reported only for PickScore and without Best-of-K.

Figure 6 and 7 examines how the performance of iterative zeroth-order methods varies with the number of particles per iteration; therefore Best-of-K cannot be included in this comparison.

  • No wall-clock time or memory is reported anywhere, even though Best-of-K is fully parallel while ZeNOVA runs K sequential rounds. The “efficient” claim is therefore unsupported.

Please refer to the table above in the responses on 6rQy and p26h that shows a similar wall-clock time and VRAM usage. And throughout this paper, the term “efficient” refers to sample efficiency, therefore characterizes ZeNOVA which achieves the highest performance under a fixed sampling budget. Meanwhile, although Best-of-K might be more “time-efficient” when one generates all samples in parallel, it require orders of magnitudes more VRAM than the other baselines, making it less “GPU-efficient”.

  • First-order baselines run 400 to 800 iterations on different hardware, which weakens the claim that they hack the reward.

Differences in hardware are irrelevant to whether reward hacking occurs. Furthermore, we have already demonstrated the reward hacking issue in first-order methods in Table 1 of the manuscript, where improvements in the optimized reward do not translate into improvements in other validation metrics that assess the same aspect of image quality.

  • Task performance on black-box rewards is measured only by the reward being optimized (besides the aesthetic score), nothing checks whether the rest of the prompt is preserved.

For the Counting task, each prompt is only "number + noun", so achieving a high reward leaves no more prompt requirements to satisfy. But since OCR prompts contain additional instructions beyond the target text, we report PickScore results for the OCR task in Table 1 of the updated manuscript. ZeNOVA achieves the highest human preference score, which also accounts for prompt alignment.

  • At the 1/λ = 1000 setting used in Table 1, ZeNOVA has the lowest Vendi score in Fig. 7.

We insist that high diversity alone without high rewards cannot be considered desirable(otherwise, the base model would trivially achieve the best diversity). Instead, the reward–diversity trade-off should be evaluated through a Pareto curve, where ZeNOVA locates the top-right corner in Figure 7.

  • The video results have no baseline beyond the base model, and Fig. 13 shows clear prompt drift.

Following the reviewer’s concern, we generally updated the results with more computational budget and report the comparison with Best-of-K and the strongest zeroth-order baseline for the video generation task in Table 2 of the updated manuscript.

  • All image results use SDXL-Turbo and SDXL-DMD2, yet evaluates none of the modern ones.

We believe that improving a poorly performing model to achieve substantial reward gain is at least as challenging as further improving a model that already performs well. That said, we additionally report the performance of ZeNOVA on Sana-Sprint(2025)[2], a more recent few-step distilled flow-matching model with a DiT architecture, in Table 1 of the updated manuscript.


[1] Gradient-Free Noise Optimization for Reward Alignment in Generative Models, arXiv 2026.

[2] SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillation, ICCV 2025

Reviewer_p26hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the challenge of aligning few-step or single-step generative models with black-box, non-differentiable reward functions, this paper proposes ZeNOVA—a method for optimizing initial noise during the inference phase. ZeNOVA relies exclusively on forward evaluations of the generator and the reward model, requiring no backpropagation whatsoever. It integrates three synergistic mechanisms: annealed soft-value guidance, hyperspherical Langevin dynamics, and Metropolis-Hastings jumps. The method has demonstrated effectiveness across various black-box and preference-based reward scenarios in both text-to-image and text-to-video generation tasks.

Critical strengths
  1. The methodology design is supported by a solid theoretical foundation. The interpolation properties of soft-value tilted distributions, the derivation of zero-order value gradient estimates, and the correctness of the Metropolis-Hastings (MH) target distribution are all rigorously proven. Furthermore, the derivation of the score function for hyperspherical/radial updates is consistent with the provided pseudocode. These elements ensure that the three components of the proposed method are not merely an empirical assembly, thereby enhancing the technical credibility of the paper.
  2. The experimental evaluation and component ablation studies are comprehensive. The proposed method was evaluated across five tasks using two different models, consistently demonstrating superior performance. Its applicability was further evidenced by its successful adaptation to a video generation model. Additionally, ablation studies on key components confirm the necessity of each design element.
Critical weaknesses
  1. The baselines and experimental statistics may be insufficient. While the most relevant gradient-free noisy optimization approach (fixed-radius projection) is cited and compared against in the 2D toy experiments, it is omitted from the comparisons in the generative modeling experiments. Furthermore, no baselines are provided for the video generation tasks. Additionally, the results lack information regarding multiple random seeds, standard deviations, confidence intervals, or error bars; given that the metrics in Table 1 are quite similar, it is possible that the observed differences are merely due to random fluctuations.
  2. Computational costs are not reported. The paper does not provide a comparison of execution times between the methods. Since the proposed approach appears to involve multiple search steps, it is unclear whether it offers any advantage in terms of computational cost. It is also necessary to report the number of queries required for both ZeNOVA and Best-of-K.
Questions for authors
  1. Please provide a comparison of the computational costs for each method, including the number of forward passes of the generator, the number of reward queries, and the total time required.
  2. Do the results in Table 1 represent the mean values across multiple random seeds? Could you also report the standard deviation, confidence intervals, or error bars?
  3. Why is the gradient-free noise optimization method proposed by Kim et al. (Gradient-free Noise Optimization for Reward Alignment in Generative Models) not included in the comparison for image generation?
  4. It is recommended to include comparisons with the latest models based on DiT (Diffusion Transformer) and Flow Matching, such as Zimage, QwenImage, and FLUX.
2 comments
AuthorsOpenReview ↗

Response to Reviewer p26h

We appreciate the reviewer p26h for the review and thoughtful feedback. Below, we provide detailed point-by-point responses to address your remaining concerns.

W1: The baselines and experimental statistics may be insufficient.

First, we would like to clarify that the baselines illustrated in the 2D toy example(Figure 3) were also included in the main experiments as ReNO(b), MoNO and ZO Exponential Map(c), ORIGEN and ZO Langevin(d), and our ZeNOVA(e). The only exception was (a), but since the output distributions shown in (a)-(c) clearly deviate from the ideal behavior, we considered this sufficient to demonstrate these empirical choices’ suboptimality. Please also refer to our response to Q3 below.

For the video generation task, we generally updated the results with more computational budget and report the comparison with Best-of-K and the strongest zeroth-order baseline in revised Table 2 of the updated manuscript.

Meanwhile, we insist that we used identical initial noise seeds across all methods to minimize factors of random fluctuations other than the methodology itself. To further support ZeNOVA's meaningful performance improvements over random chance, we now additionally evaluated validation metrics that measure the same aspects of quality as the corresponding rewards for all tasks except Counting. ZeNOVA maintained its superior performance under these validation metrics as well.

W2: Computational costs are not reported.

We would like to clarify that as stated in Lines 967-971, all first-order/zeroth-order methods and Best-of-K used the same number of generated image samples(400 or 800), and the number of generator passes and reward queries naturally followed this number. As a result, all zeroth-order methods exhibited similar runtimes. Please refer to the table below for a comparison of wall-clock time and VRAM usage.

Required time[s] / allocated VRAM[GB]ZeNOVAZeNOZO LangevinZO Exponential MapNESBest-of-K
ImageReward (400 images)43.992 / 15.9341.686 / 15.9340.949 / 15.9341.644 / 15.9341.223 / 15.9337.249 / 15.93
PickScore (800 images)93.918 / 18.0087.834 / 18.0087.575 / 18.0087.657 / 18.0187.432 / 18.0179.104 / 18.00
GenEval (800 images)208.763 / 14.26195.556 / 14.26199.769 / 14.26194.806 / 14.26199.358 / 14.26188.742 / 14.26
Counting (400 images)67.953 / 18.3565.171 / 18.3565.012 / 18.3465.394 / 18.3465.142 / 18.3560.478 / 18.34
OCR (800 images)150.704 / 14.25136.191 / 14.25136.755 / 14.27146.901 / 14.26141.157 / 14.27129.790 / 14.27

Q1: Please provide a comparison of the computational costs for each method, including the number of forward passes of the generator, the number of reward queries, and the total time required.

Please refer to our response in W2 above.

Q2: Do the results in Table 1 represent the mean values across multiple random seeds? Could you also report the standard deviation, confidence intervals, or error bars?

Please refer to our response in W1 above.

Q3: Why is the gradient-free noise optimization method proposed by [1] not included in the comparison for image generation?

Thank you for pointing it out. We had examined the suggested baseline and found that, although [1] claims to use the value gradient, their use of several overly strong assumptions(linear reward assumption, 1-step horizon, etc.) ultimately reduces their algorithm to naive ZO Langevin dynamics (see Algorithm 1 of [1] and our Eq. 7), but with an additional norm projection to d\sqrt{d}. We further emphasize that [1] doesn’t explain how value gradient alters the sampling distribution and the benefits it provides, as we establish in Proposition 1. Nevertheless, we provide a comparison with [1] in Table 1 of the updated manuscript, where ZeNOVA continues to outperform the baselines. We note that [1] originally used 3200 samples per prompt, where in our experiment it’s much less(400 or 800). This demonstrates that the naive norm fix(Figure 3-(a)) is a suboptimal choice for Gaussian prior preservation.

Q4: It is recommended to include comparisons with the latest models based on DiT (Diffusion Transformer) and Flow Matching, such as Zimage, QwenImage, and FLUX.

We note that rCM for text-to-video generation employs a DiT architecture. Furthermore, since the methodology of ZeNOVA does not rely on any specific architectural properties, we claim its performance is model architecture-agnostic. That said, we additionally report the performance of ZeNOVA on Sana-Sprint[2], a more recent few-step distilled flow-matching model with a DiT architecture, in Table 1 of the updated manuscript.


[1] Gradient-Free Noise Optimization for Reward Alignment in Generative Models, arXiv 2026.

[2] SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillation, ICCV 2025

AuthorsOpenReview ↗

Reminder for the discussion period deadline

Dear Reviewer p26h,

We are writing to kindly remind you regarding our responses to your comments on our manuscript. As the deadline for the discussion is approaching within 24 hours, we would greatly appreciate it if you could timely consider our revision and inform us of your decisions. Please let us know if there are any further clarifications needed.

Best regards, Authors

Reviewer_6rQyOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors aim to solve the problem of black-box reward alignment via initial noise optimization for flow-map based/distilled flow-based models. They first identify three common pitfalls in zeroth order initial noise optimization: nondifferentiability that causes difficulties to explore, destabilization from few-particle estimates and hazard of bad initial states. They then propose a solution for each of these problems: annealed soft-value guidance, manifold-constrained hyperspherical Langevin dynamics, and Metropolis-Hastings jumping. Together, they propose ZeNOVA, and validate its advantages against baselines on various distilled models and tasks.

Critical strengths
  1. This is a very well written paper and all parts of the paper are really well connected. I really like how the authors motivate the problem, introduce the most naive solution, identify the failure modes, propose solutions for each failure mode, and then present experimental evidence on how their solutions are effective. Whenever I have a question regarding the paper, it will get immediately answered by the content following.
  2. Each solution the authors propose is well thought-out and grounded. I particularly like the manifold-constrained hyperspherical Langevin dynamics. It is based on a commonly known yet under-explored insight and it is extremely effective. Super clever.
  3. The experiments are also very convincing and comprehensive. The ablation study is clear and compelling.
Critical weaknesses

I don't really see any critical weakness in this paper to be honest. I do have a few questions which I will include below.

Questions for authors
  1. How do the baseline vs ZeNOVA compare in terms of speed? (I know that in Figure 6 the authors have provided particle count efficiency comparison, but how do all baselines vs ZeNOVA compare in terms of wall clock time?)
  2. From Algorithm 1, it seems like you always perform multiple MH transitions every time you perform Langevin updates, but this looks a bit contradicting to your description in line 280-282. How exactly does MH jump work? Is it applied more frequently at the beginning of the optimization or it's frequency is a constant throughout the optimization process. How do you decide how often to perform MH jump?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 6rQy

We thank the reviewer 6rQy's thoughtful feedback and for considering our work. Below we provide point-by-point responses on your questions and concerns.

Q1: How do the baseline vs ZeNOVA compare in terms of speed? (I know that in Figure 6 the authors have provided particle count efficiency comparison, but how do all baselines vs ZeNOVA compare in terms of wall clock time?)

Please refer to the table below, which contains the required time and allocated VRAM per image for each method; when the total number of particles was fixed, although ZeNOVA admits slightly more time, all methods had similar required runtime. Since the proposal distribution of MH jumping is independent of the current z, we can batchify all image sampling operations in a single ZeNOVA iteration, preventing a substantial runtime increase.

Required time[s] / allocated VRAM[GB]ZeNOVAZeNOZO LangevinZO Exponential MapNESBest-of-K
ImageReward (400 images)43.992 / 15.9341.686 / 15.9340.949 / 15.9341.644 / 15.9341.223 / 15.9337.249 / 15.93
PickScore (800 images)93.918 / 18.0087.834 / 18.0087.575 / 18.0087.657 / 18.0187.432 / 18.0179.104 / 18.00
GenEval (800 images)208.763 / 14.26195.556 / 14.26199.769 / 14.26194.806 / 14.26199.358 / 14.26188.742 / 14.26
Counting (400 images)67.953 / 18.3565.171 / 18.3565.012 / 18.3465.394 / 18.3465.142 / 18.3560.478 / 18.34
OCR (800 images)150.704 / 14.25136.191 / 14.25136.755 / 14.27146.901 / 14.26141.157 / 14.27129.790 / 14.27

Q2: From Algorithm 1, it seems like you always perform multiple MH transitions every time you perform Langevin updates, but this looks a bit contradicting to your description in line 280-282. How exactly does MH jump work? Is it applied more frequently at the beginning of the optimization or it's frequency is a constant throughout the optimization process. How do you decide how often to perform MH jump?

Thank you for pointing this out. The implementation for the main results in the manuscript uses a fixed number of MH jumps at every iteration, as described in Algorithm 1. Lines 202-282 simply explain that its effectiveness is more focused in the early stage of the optimization process, as more MH jumping proposals will be rejected as optimization progresses(Lines 277-278).

Thank you for your rebuttal! It has addressed my questions so my score will remain the same.

Thank you for your encouraging feedback. We are glad that our responses have addressed your questions, and we sincerely appreciate your continued positive assessment of our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Reviewers

We sincerely thank all the reviewers for their review and thoughtful feedback, and for recognizing the strengths of our work, particularly its clear motivation and presentation [6rQy, nPyD], theoretically grounded methodology [6rQy, p26h], effective component design [6rQy, p26h], and comprehensive experimental evaluations and ablation studies [6rQy, p26h]. We have addressed the reviewers' concerns in our detailed responses, with the following key clarifications and additional experiments:


  1. Necessity of all three components and consistent empirical superiority: We clarify that all component is essential to ZeNOVA's applicability across diverse black-box reward scenarios. Removing an individual component may yield competitive performance on certain tasks, but it eventually compromises performance on others. Moreover, ZeNOVA consistently achieves the best performance across all evaluated models and tasks; we believe that relatively small gaps on certain metrics don’t undermine its overall performance.
  2. Expanded baseline comparisons and validation metrics: We additionally evaluate ZeNO [1] on text-to-image models and demonstrate that ZeNOVA maintains superior performance, and introduce additional validation metrics to further assess reward alignment and prompt fidelity. We include Best-of-K and the strongest zeroth-order baseline for the text-to-video task, providing more comprehensive evidence of ZeNOVA's effectiveness beyond image generation.
  3. Evaluation on a more recent generative model: We extend our experiments to Sana-Sprint, a recent few-step distilled flow-matching model employing a DiT architecture, to further demonstrate the applicability of ZeNOVA to modern generative architectures.

We also provide additional analyses, including wall-clock time and VRAM comparisons, multiple-seed evaluations, and further ablation studies. Please refer to the individual responses for each reviewer below for more detailed responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.