acceptodds
Under review as a conference paper at ICLR 2027

Noise-Space Reward Alignment for Few-Step Generative Models

Abstract

Few-step generative models produce high-quality samples in one or a few evaluations. However, steering them toward desired properties remains difficult, especially when only black-box reward evaluations are available. While one could apply approaches developed for iterative diffusion and flow-based models, such as direct RL fine-tuning or test-time scaling, we find that neither transfers well to the few-step regime: direct fine-tuning suffers from severe reward over-optimization, and test-time scaling sacrifices the efficiency advantage of few-step generation. We propose Noise Policy Optimization (NPO), which explicitly targets the alignment problem for few-step generative models. The key idea is to learn an amortized flow policy over the initial noise space of the generator. The trained policy transports Gaussian noise to a reward-tilted noise distribution, whose pushforward distribution through the frozen generator is exactly the reward-tilted output distribution. By keeping the generator frozen, we can effectively utilize the knowledge encoded in the pretrained model, yielding a better reward-quality trade-off than direct fine-tuning. The amortized policy moves the search cost into training, so inference adds only a lightweight noise sampling step on top of the pretrained model. To improve sample efficiency during training, we use ELBO-based likelihood estimation that requires only sampled endpoints rather than simulated trajectories. Across image, language, and protein structure generation, NPO consistently achieves a better reward-quality trade-off than direct fine-tuning, and matches test-time search methods with 16 to 549x lower inference cost.

Primary area:
generative models
Keywords:
Few-step Generative Model, Reward Alignment, Flow-based Generative Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_akQFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper aligns few-step, deterministic generators under black-box rewards by freezing the generator and moving alignment into its input space. Sampling from the reward-tilted output reduces to sampling tilted noise, which NPO learns with a flow policy trained to minimize KL(qψ∥q⋆)\mathrm{KL}(q_\psi\|q^\star) under reference regularization toward qbaseq_{\text{base}}; the policy gradient uses an ELBO-based endpoint likelihood from MM forward-noising pairs rather than simulated trajectories. Validation covers text-to-image (SD-Turbo, SANA-Sprint), topic-steered language (4-step ELF) and protein backbones (SiD-Protein) against direct fine-tuning, test-time search and trajectory-based noise policies, claiming a better reward-quality trade-off than fine-tuning and comparable reward to test-time search at 16–549× lower cost.

Critical strengths
  • The reformulation is exact for deterministic generators and converts an intractable likelihood problem into a standard flow-fitting one, which is what makes black-box rewards and deterministic samplers share one machinery.
  • The empirical design is broad and includes the right control: three modalities, four generators across 1/4/5 steps, and a baseline differing only in the likelihood estimator. The cost tables are the correct way to argue against test-time search.
Critical weaknesses
  • The importance ratio in Eq. (7) is not a valid density ratio. ℓψ\ell_\psi is a single-pair residual, Eq. (6) holds only in expectation, and exp⁡(ℓold−ℓ)\exp(\ell_{\text{old}}-\ell) on one shared pair is neither unbiased nor consistent; with M=3M=3 the averaging is thin. It is unclear whether this is a heuristic weight (then say so) or a correction (then it needs justification and clipping).
  • The headline negative claim rests on baselines not shown to be tuned. Flow-GRPO drives protein designability to exactly zero, which reads as a mis-specified baseline (learning rate, KL coefficient, injected stochasticity) rather than a property of the method class. Without a best-effort baseline, "direct fine-tuning transfers poorly" is not established.
Questions for authors
  1. Is ρ^\hat\rho clipped, and what happens at M=1M=1? Please justify it as a ratio or state that it is a heuristic weight.
  2. What sweep was run for Flow-GRPO and FAV-B, and is the designability collapse reproduced across seeds and learning rates?
  3. Appendix D.3 says five seeds with mean and standard deviation; why do Tables 1–2 report point estimates? Please add them, and give the image results as a table rather than only curves.
2 comments
AuthorsOpenReview ↗

Rebuttal response to Reviewer akQF

We thank the reviewer for the detailed examination of our method and experiments and the helpful suggestions. We would like to address their concerns and questions below.

(W1 & Q1) Clarifying the importance ratio approximation and implementation.

Thank you for raising this point! For the importance ratio, our formulation follows [1]. Specifically, we use the ELBO-based ratio approximation and construct ρ^ψ\hat\rho_\psi from the difference between regression losses. This is a surrogate weight for the policy update rather than a guaranteed unbiased or consistent estimator of the exact density ratio. We have updated our paper to clarify this heuristics.

We omit ratio clipping and advantage standard-deviation normalization for methodological simplicity and fewer hyperparameters. Figure 15 in Appendix H.2 compares these choices and shows comparable performance with and without clipping and normalization.

Figure 11 in Appendix H.2 compares M=1,3,6M=1,3,6 forward-noising pairs. Increasing MM beyond three provides no observed gain in this experiment, so we use M=3M=3 to balance performance and computational cost.

[1] Xue, et al. “Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models.” ICML, 2026.

(W2 & Q2) Addressing fair comparison concerns related to hyperparameter tuning.

We carefully tuned the hyperparameters for all methods, including baselines. Specifically, we excluded configurations where the target reward dropped abruptly or consistently decreased during training. Among the remaining configurations, we select the best setting whose target reward rises stably across 3 different seeds. We use the same strategy to select this hyperparameter for all methods, including our NPO. We use 3 seeds to select the hyperparameters and run two additional seeds for the selected ones. These selection criteria are detailed in Appendices D, E, and F.

Both Flow-GRPO and FAV-B were tuned over the learning-rate and regularization-strength ranges reported in Table 5-11 in Appendix D,E,F. To make the comparison more explicit, we have added reward–quality trade-off plot across regularization strengths averaged by 3 seeds for both NPO, Flow-GRPO and FAV-B (Figures 17, Appendix H.3). The results consistently support the same conclusion that NPO achieves a better reward–quality trade-off than direct fine-tuning.

(Q3) Standard deviations and image results as tables.

Thank you for pointing this out! In our initial draft, experiments for test-time optimization comparison (Table 1, 2, Fig 7.a) were originally reported using a single seed. We sincerely apologize for this inconsistency. We have now updated Tables 1–2, Fig 7.a to report mean ± standard deviation over five seeds and corrected the experimental details accordingly. Our conclusion remains the same with these new results.

In Appendix G, we also provide numerical tables for the image results in the main paper, with mean ± standard deviation for each metric.

Thank you again for your detailed review and valuable suggestions. We hope our responses have addressed your concerns. Please let us know if you have any further questions.

Reviewer_PokZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Noise Policy Optimization (NPO), an algorithm to align few-step flow/ consistency models (deterministic) to black-box rewards while keeping the pre-trained generator frozen. For deterministic samplers, the optimal distribution corresponding to the entropy regularized reward maximization objective is fully specified by a corresponding tilted initial noise distribution. The algorithm proposes to essentially learn this tilted initial noise distribution by training a separate flow model. Empirical results are given on image, language and protein structure generation.

Critical strengths

(1) NPO trains a new flow model to sample from the optimal tilted initial noise distribution; since the actual pre-trained model is left untouched, the algorithm is potentially more stable compared to other methods which often have to fine-tune a pre-trained model and it shows a better reward–quality trade-off empirically. Further, gradients only flow through the small noise policy, which avoids backpropagating through the large pre-trained model during training (although the noise policy does increase inference cost).

(2) Experiments cover three domains showing that the algorithm generalizes across domains and is not specific to say image generation.

Critical weaknesses

(1) The idea that, for a deterministic sampler, one can get away with learning only an optimal initial noise distribution is already known in literature. For example, [1] show that entropy-regularized fine-tuning requires tilting the initial distribution by the soft value function; for deterministic generators this coincides with the tilted noise distribution used here. Similarly, Noise Hypernetworks [2] also learns the same tilted noise distribution for few-step generation (although in white-box setting). Given this, the primary novelty seems to be the use of an ELBO-based policy-gradient objective to learn this noise distribution. While arguably novel, the overall novelty still seems limited.

(2) Given (1), it would help if the baseline comparisons are a bit more comprehensive. For example, the only direct fine-tuning methods which are compared to seem to be Flow-GRPO and FAV-B. Could the authors compare against ELBO-based direct fine-tuning approaches like say AWM [3] or DiffusionNFT [4] wherever they are applicable, since the paper’s own comparisons seem to favor ELBO based methods?

[1] Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control arxiv:2402.15194

[2] Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models, arxiv:2508.09968

[3] Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models, arxiv:2509.25050

[4] DiffusionNFT: Online Diffusion Reinforcement with Forward Process, arxiv:2509.16117

Questions for authors

Any clarifications regarding novelty or additional comparisons addressing weakness (2) would be helpful.

2 comments
AuthorsOpenReview ↗

Rebuttal response to Reviewer PokZ

We thank the reviewer for the thoughtful assessment of our work and the constructive suggestions for strengthening the paper. We would like to answer their questions and concerns below.

(W1) Clarifying novelty and positioning relative to prior work.

Our main contribution is to introduce an effective and efficient method for aligning few-step generative models without requiring reward gradients. As discussed in the introduction and Sections 2–3 of our original submission, existing approaches face several challenges in this setting. In particular, direct fine-tuning methods struggle with the reward–quality trade-off, test-time optimization methods incur high inference costs, and trajectory-based noise-policy learning methods suffer from training inefficiency. Our NPO is specifically designed for this setting to address these challenges.

We fully acknowledge that the tilted initial-noise distribution has been established in prior work, which we have discussed in Sections 2–3 of our original submission. We also appreciate the additional references provided by the reviewer and would like to further clarify our positioning relative to these works. ELEGANT [1] controls continuous-time diffusion dynamics and does not directly applicable to the few-step generators considered in our paper. Noise Hypernetworks [2], which we have already included in our original related wrok discussion, addresses few-step generators but requires reward gradients, making it inapplicable to our black-box reward settings, including JPEG compressibility, language, and protein generation. The table below summarizes these distinctions.

MethodFew-step generative model alignmentWithout reward gradients
ELEGANT [1]✗✗
Noise Hypernetworks [2]✓✗
NPO (Ours)✓✓

We expect alignment of few-step generators with black-box rewards to become increasingly important as few-step generators improve and advances in simulators and LLM-based evaluation provide increasingly useful reward signals. We have included this extended discussion of related work and citations to these works in our updated paper.

(W2) Extending comparisons with ELBO-based direct fine-tuning methods.

Thank you for pointing this out! We agree with the reviewer that including the comparison with ELBO-based direct fine-tuning methods can significantly strength our paper. Therefore, we implemented AWM [3] and DiffusionNFT [4] as heuristic baselines under the additional assumption that the outputs of few-step generative models can be treated as instantaneous velocities, and report the results in Figure 19 of Appendix H.4. Both exhibit deterioration in sample-quality and diversity metrics as the target reward increases. NPO maintains a better reward–quality trade-off in these comparisons, supporting our conclusion against the evaluated direct fine-tuning baselines.

Thank you again for your detailed review and valuable suggestions. We hope our responses have addressed your concerns. Please let us know if you have any further questions.

Reviewer_XHYpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper learns a flow model that samples the initial noise of a pretrained few-step generator, in order to improve reward alignment. The authors aim to maximise a reward objective regularised by the KL divergence between the initial noise distribution and the learned adjusted distribution. The KL term is approximated via a squared deviation of the learned flow to a a reference, and the reward term is optimised using advantages and an estimate of the log likelihood. The authors demonstrate performance in text-to-image generation, language generation, and protein structure generation.

Critical strengths

The paper is clearly written, and I found it straightforward to follow. I appreciate that the authors have evaluated on a range of different data modalities, and for comparing to examples from a range of categories of alignment methods (namely 'direct generator fine-tuning', 'test-time noise optimisation', and 'noise-policy learning', which I feel do a good job of covering the various different approaches for this problem). I also thank the authors for including a large amount of detail in the appendix regarding experimental setups.

Critical weaknesses

Many of the design decisions in the proposed method are only loosely grounded in theory. The log-likelihood and the KL divergence are both estimated according to integrated squared differences in the vector field, and I think it would be instructive to explain in the main paper why these expressions help to estimate the terms in question (to make easier for the reader to understand). Moreover, the actual implementation of the likelihood surrogate described in the appendix normalises the squared term, making it further removed from the theoretical justification. While I do not consider this somewhat weak theoretical grounding a major obstacle provided the empirical performance is strong, I nevertheless think the paper would benefit from a discussion of these points in the main body.

I think the way in which the regularisation strengths are chosen make it unclear how to interpret the reported metrics. The appendix states that the regularisation strengths and learning rates were swept, and then the metrics are reported for the settings that caused the target reward to rise stably across seeds (I find this description a bit vague). This makes it hard to compare between the methods - the rewards can generally be made higher just by reducing the amount of regularisation, so it is not clear whether the differences in performance are caused by the methods themselves, or by regularisation strengths that are not comparable between methods.

Related to this, I also find it surprising that the results in Figure 12 seem to suggest that the regularisation strength has minimal effect on the results (indeed, the method seems to perform very similarly even with the regularisation turned off). This somewhat undermines the argument for needing any regularisation at all, and also means that we don't really know whether the regularisation term is doing anything (even though it is presented as a crucial part of the proposed approach). I think the paper is missing an experiment that demonstrates that the proposed regularisation term is necessary.

Questions for authors

Please see my points above. I also have the following additional questions.

The authors comment that finetuning the generative model itself can be difficult as they either struggle to learn, or suffer from large amounts of reward hacking. However, by keeping the generative model fixed, this may limit the amount of reward alignment that can be attained, as the generations will remain in the support of the original generative model. Do the authors have any comments regarding this as a potential limitation? (I realise is shared with other reward-alignment methods, but I think it is an important factor that could be discussed as it helps place the proposed method relative to finetuning approaches).

Have the authors tried out their approach in settings where reward gradients are available? For example, they are available in the text-to-image setting. This would help to assess how effective the KL regularisation term used in the procedure is (as reward gradients give strong signal, so require effective regularisation techniques).

4 comments
AuthorsOpenReview ↗

Rebuttal response to Reviewer XHYp (1/2)

We thank the reviewer for their careful reading of our paper and their insightful feedback. We would like to address the reviewer's concerns and questions below.

(W1) Improving theoretical grounding and discussions.

We appreciate the reviewer's feedback and recognition of our strong empirical performance. To address their feedback, we would like to first highlight the theoretical grounding already provided in our original submission, and then introduce additional theoretical connections in our revision (marked in red).

In particular, in Appendix B.2 of our original submission, we have motivated the squared velocity-difference approximation for KL regularization through the Girsanov relation between path-space KL and quadratic drift costs [3]. In Appendix B.3 of our revision, we further clarify the connection between our weighted flow matching likelihood surrogate and the diffusion ELBO under appropriate time weighting [1,2].

To strenghthen our theoretical grounding, we have added clarifications for these connections in the appendix and added explicit references to Appendices B.3 and B.2 in Section 4.4 for the likelihood estimator and KL regularization, respectively.

Regarding adaptive normalization, it is indeed a practical weighting choice for numerical robustness, as suggested in [4]. Figure 10 in Appendix H.2 compares adaptive weighting with path-KL and simple weighting, showing its favorable empirical performance.

[1] Song, et al. “Maximum Likelihood Training of Score-Based Diffusion Models.” NeurIPS 2021.

[2] Kingma, and Ruiqi Gao. “Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation.” NeurIPS 2023.

[3] Choi, et al. “Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design.” ICML 2026.

[4] Zheng, et al. “DiffusionNFT: Online Diffusion Reinforcement with Forward Process.” ICLR 2026.

(W2) Addressing fair comparison concerns related to regularization hyperparameter tuning.

We carefully tuned the hyperparameters for all methods, including baselines. Specifically, we excluded configurations where the target reward dropped abruptly or consistently decreased during training. Among the remaining configurations, we select the best setting whose target reward rises stably across 3 different seeds. We use the same strategy to select this hyperparameter for all methods, including our NPO. We use 3 seeds to select the hyperparameters and run two additional seeds for the selected ones. These selection criteria are detailed in Appendices D, E, and F.

To further ensure fair comparisons, we have added reward–quality trade-offs for direct fine-tuning baselines and training curves for noise-policy baselines across regularization strengths averaged by 3 seeds (Figures 16–18, Appendix H.3). The results consistently support the same conclusion that NPO achieves a better reward–quality trade-off than direct fine-tuning and more efficient reward optimization than trajectory-based noise-policy baselines.

(W3) Effect of regularization.

To further examine the effect of regularization, we extend the ablation in Figure 12 in Appendix H.2 to include larger regularization strengths and add ablation results on language tasks in Figure 13. The expanded sweep shows clearer regularization effects: stronger regularization lowers the final target reward, while keeping the evaluated sample-quality metrics closer to those of the pretrained model.

In our image experiments, both target reward and sample-quality metrics improve together, so stronger regularization provides limited benefits on these metrics. In our language experiments, however, diversity decreases as the target reward increases. Stronger regularization preserves sample diversity closer to that of the pretrained model, allowing a preference for more diverse samples.

AuthorsOpenReview ↗

Rebuttal response to Reviewer XHYp (2/2)

(Q1) "Will frozen generator potentially limit attainable alignment?"

We fully agree with the reviewer and acknowledge this potential limitation. NPO targets p∗(x)∝pref(x)exp⁡(r(x)/τ)p^*(x)\propto p_{\mathrm{ref}}(x)\exp(r(x)/\tau), the reward-tilted distribution that is commonly used in reference-regularized generative-model alignment. This target is indeed only supported on the reference distribution. However, as the reviewer has also mentioned, this support restriction is shared by all methods targeting this distribution, including many of our baselines, rather than being unique to NPO.

We agree that discovering high-reward samples outside the frozen generator’s support is an important direction for future research. Nevertheless, we expect noise-space alignment to become increasingly attractive as pretrained generators improve. As larger and more diverse pretraining datasets bring more high-reward samples within their generative capabilities, NPO can effectively leverage this knowledge, remaining a computationally efficient and effective alternative to direct fine-tuning that scales with pretrained models. We have included our discussion here in the limitation section of the paper.

(Q2) Experiments where reward gradients are available.

Our text-to-image experiments already include a differentiable aesthetic reward. However, NPO treats it as a black box and uses only scalar evaluations, without accessing its gradients.

Our focus is to align few-step generators without requiring reward gradients, which may be unavailable for programmatic rewards, simulators, external APIs, or discrete operations during generation or reward evaluation. As simulators and LLMs improve, we expect such black-box rewards to provide increasingly useful signals, making gradient-free alignment a promising direction.

Thank you again for your detailed review and valuable suggestions. We hope our responses have addressed your concerns. Please let us know if you have any further questions.

Reviewer_XHYpOpenReview ↗

Acknowledgement of rebuttal

I thank the authors for their rebuttal, and it has addressed many of my concerns. I have raised my score as a result.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to reviewers

We sincerely thank all reviewers for their thoughtful and constructive comments. We appreciate their recognition of our paper's strengths, including

  1. Broad empirical evaluation across three modalities (XHYp, PokZ, akQF)
  2. Comprehensive baseline comparisons (XHYp, akQF)
  3. Effective and stable alignment with a frozen generator (PokZ)
  4. Efficient training without backpropagating through the pretrained generator (PokZ)
  5. Clear presentation and detailed experimental setups (XHYp)

In response to the reviewers' feedback, we have conducted additional experiments and updated our paper, as summarized below.

Additional Experiments

  • ELBO-based direct fine-tuning baselines: We additionally evaluate AWM and DiffusionNFT on few-step generators (Figure 19, Appendix H.4).
  • Effect of regularization: To further examine the effect of regularization, we extended our experiments to larger regularization strengths and added language experiments (Figures 12–13, Appendix H.2).
  • Multi-seed results for test-time optmization comparison: We include the 5-seed results for test-time optimization comparison experiment to keep our experimental setting consistent across the board. (Table 1, 2, Fig 7.a)

Paper Revisions

  • Theoretical grounding: We further clarify the connections between our weighted flow-matching likelihood surrogate and the diffusion ELBO, and between KL regularization and the Girsanov relation (Section 4.4, Appendix B).
  • Importance-ratio approximation: Following prior works, we clarify that the ELBO-based likelihood-ratio approximation serves as a heuristic surrogate (Section 4.4).
  • Hyperparameter selection and comparisons: We clarify our selection criteria and provide reward–quality trade-offs and training curves across regularization strengths (Figures 16–18, Appendix H.3).
  • Additional numerical results: We report mean ± standard deviation over five seeds in Tables 1–2 and provide tabulated image results in Appendix G.
  • Related work and positioning: We expand the related-work discussion to clarify NPO's positioning relative to ELEGANT and Noise Hypernetworks (Sections 2).
  • Limitations: We further discuss the support restriction of frozen generators and future directions for noise-space alignment.

We sincerely thank the reviewers again for their constructive feedback and hope these additions and revisions address their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.