acceptodds
Under review as a conference paper at ICLR 2027

Learning Noise Optimization for Reward Alignment in Few-Step Generative Models

19 views

Abstract

Reward alignment for diffusion- and flow-based generative models is often formulated over stochastic trajectories, making it less suitable for few-step deterministic generators. Initial noise optimization provides a trajectory-free alternative, but existing gradient-free methods require costly per-sample search at inference. We introduce LeNO (Learning Noise Optimization), which amortizes reward-guided initial noise optimization into a learned residual transport while keeping the pretrained generator frozen. LeNO learns an anchor-conditioned distribution over noise corrections using Local GRPO and scalar rewards, eliminating iterative test-time optimization. For Gaussian initial noise with norm-preserving projection, we derive a tractable upper bound on the KL divergence between the corrected noise distribution and the original prior, providing a principled regularization objective. We evaluate LeNO on few-step text-to-image and protein structure generation, including expensive black-box designability rewards. Across both domains, LeNO improves target rewards with a single noise correction at inference while largely preserving generation quality and diversity.

Primary area:
generative models
Keywords:
Noise Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_cVQvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a training-time approach to initial noise optimization for deterministic generative models. The authors identify that existing test-time approaches require iterative refinement, requiring many generator and reward evaluations. To avoid this cost, they train a residual transport map using a local GRPO objective, resulting in reward-aligned samples with a single inference step. To mitigate reward over-optimization, they introduce a KL-based regularizer that constrains the corrected-noise distribution to remain close to the original prior. The method is evaluated on a toy problem, text-to-image generation, and protein structure generation. Results suggest that LeNO improves the target rewards while requiring substantially fewer inference-time function evaluations than iterative noise-optimization baselines.

Critical strengths
  1. The evaluation suite is strong. The authors compare against both amortized training-time and iterative inference-time baselines across diverse settings, including text-to-image generation with differentiable preference rewards and protein structure generation with an expensive non-differentiable reward. This breadth helps contextualize the method’s advantages.

  2. The empirical results are strong. LeNO consistently improves target rewards over the frozen text-to-image models. Although iterative inference-time baselines achieve higher absolute reward when given substantially larger search budgets, combining LeNO with Best-of-N yields competitive or stronger results using far fewer NFEs. This demonstrates a meaningful inference-efficiency tradeoff rather than simply reporting reward metrics in isolation.

Critical weaknesses
  1. LeNO trades expensive inference-time refinement for expensive pre-inference training. Although the paper specifies training hyperparameters, the authors do not directly report the computational overhead for training this transport. This omission make it difficult to assess the method's practical advantage over inference-time approaches.
Questions for authors
  1. Can you report the computational overhead for the text-to-image and protein experiments, including wall-clock training time, number and type of GPUs, and total generator/reward evaluations? This would help clarify the practical cost of amortizing optimization at train-time instead of test-time.
4 comments
AuthorsOpenReview ↗

Response to Reviewer cVQv

We thank the reviewer for the valuable feedback on training cost and computational efficiency. We provide additional runtime measurements and an amortized cost analysis to clarify when LeNO offers practical advantages over inference-time optimization. Detailed results and explanations are provided below.

W1 & Q1 : Training Cost

We profiled LeNO on eight RTX 4090 GPUs (24 GB each), measuring six image-training iterations and five protein-training iterations after warm-up. Initialization, periodic validation, and checkpoint saving are excluded.

Model / rewardTraining stepsSeconds/stepEstimated hoursGPU-hoursReward evaluations
SDXL-Turbo / ImageReward1,00032.328.9871.8512,000
SDXL-Turbo / ImageReward5,00032.3244.89359.12,560,000
RMF-M / designability1,00013.463.7429.936,000
RMF-L / designability1,00020.505.6945.536,000

Each protein reward evaluation uses four designed sequences, totaling 144,000 ProteinMPNN sequence designs and 144,000 ESMFold refoldings per model. Counts refer to generated samples, rather than batched calls; timing includes transport forward/backward computation. RMF-L's estimate is more variable: the eight-GPU-equivalent iteration times range from 12.93 to 45.82 seconds, with the delayed iteration retained in the mean.

On the same 100 prompt–seed pairs, 1,000-step LeNO achieves similar ImageReward to five-step ZeNO (0.970 vs. 0.961), with much faster generation (0.304 vs. 7.845 s/image on one RTX 4090). Including LeNO's estimated 71.8 GPU-hours of training, cumulative compute breaks even at approximately 34,300 images. LeNO therefore benefits repeated generation under a fixed reward objective; ten-step ZeNO can achieve higher reward (1.056) at greater inference cost. We will add these costs and amortization curves to the revision.

The amortization benefit is particularly relevant for protein generation, where reward evaluation requires costly ProteinMPNN sequence design and ESMFold refolding. LeNO incurs these evaluations during training but requires no reward calls at inference. Although inference-time methods can parallelize samples and candidate evaluations, successive refinement iterations remain sequential within each sample. LeNO replaces this iterative search with a learned transport that supports batched generation, making amortization especially attractive when reward evaluation dominates computational cost.

AuthorsOpenReview ↗

A Kind Reminder for Comments

We sincerely thank Reviewer cVQv for the constructive feedback regarding training cost and the practical benefits of amortized noise optimization. Our response and revised manuscript address these concerns through the following analyses and clarifications:

  • Training cost: Added computational cost estimates for SDXL-Turbo and protein generation, including wall-clock training time, GPU-hours, and reward-evaluation budgets (Appendix D.3).
  • Amortization efficiency: Clarified the trade-off between training cost and inference efficiency. LeNO achieves comparable reward performance to inference-time ZeNO with substantially faster generation, making amortization beneficial for repeated generation under a fixed reward objective, particularly when reward evaluations are expensive.

We appreciate the reviewer's time and would welcome any remaining questions or concerns regarding these revisions. Changes in the revised manuscript are highlighted in blue.

I thank the authors for their thoughtful feedback to my concerns and their edits to the manuscript. Overall, I feel that my concerns have been addressed and I maintain my positive score.

We thank the reviewer for their careful consideration of our rebuttal and are glad that their concerns have been addressed.

Reviewer_JWzhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to learn an amortised residual correction to the sampled initial noise, which is then fed into a few-step generative model, in order to improve reward alignment. They parameterise the residual correction via a neural 'mean prediction' function (that depends both on the condition cc and the 'base' sampled noise zz), which is then used to adjust the initial noise zz before being used as input to the pretrained generative model. Through the Gaussian parameterisation, the authors derive a tractable upper bound on the KL divergence between the initial and adjusted noise distributions, which results in a regularised reward maximisation problem. The reward term in the objective is approximated using a local GRPO objective. The procedure is demonstrated in a toy experiment, and performance is examined in both image and protein generation tasks.

Critical strengths

The paper is well written, and I enjoyed reading it. The various pieces of the method are in my opinion put together in a sensible way, with each design decision well-motivated by the design of the algorithm (e.g. using the Gaussian parameterisation so that the upper bound is tractable, adding the exploration noise for the GRPO step etc). Visually, the performance appears strong. I also appreciate that the authors have reported ablations over important design choices such as the state-dependence of the mean-adjustment function μθ\mu_\theta, and the exploration scale σ\sigma.

Critical weaknesses

I think the paper does not make clear to what extent the behaviour of the approach depends on the regularisation strength, which in particular makes it difficult to compare some of the numerical results between the different methods that are reported. Reporting the values of various rewards alone is not really enough to quantitatively assess such results, as these can often be made higher by just reducing the strength of the regularisation. I am therefore somewhat unclear as to how much information one can glean from the reported attained rewards in the experimental sections, as it might be that the amounts of regularisation are not comparable between different methods. Several similar papers (e.g. [1], [2]) in this area often report curves indexed by regularisation/reward strength (and also how this transfers to difference reward functions), which in my opinion is more effective at conveying the differences between different methods as they demonstrate the reward-regularisation tradeoff which is otherwise not clear from the reported metrics.

[1] Liu et al 2025, Value Gradient Guidance for Flow Matching Alignment

[2] Zhang et al 2024, Confronting Reward Overoptimization for Diffusion Models

Questions for authors

Do the authors have any comments on how to interpret the numerical results, given they will be highly dependent on the choice of regularisation strength? (I see the KL regularisation ablation in the Appendix, but nevertheless I think this should be addressed in the main body where the values are reported)

An alternative to learning the initial noise would be to finetune the generative model itself. Several works have looked at this instead, but the trade-offs relative to such methods are not really discussed. Do the authors have any thoughts on how the proposed method differs relative to methods that optimise the generative model directly? My concern might be that optimising the noise condition only, while keeping the generative model fixed, may limit the amount of reward alignment that can be attained, as the generations will remain in the support of the original generative model. It might require different amounts of optimisation too. I appreciate that this aspect is shared by other noise-optimisation works, but in my mind it is the main question that I have about this line of work and thus I think it is worthy of discussion.

Is the Gaussian parameterisation chosen in order to ensure that there is a tractable upper bound available? Did the authors consider any alternative parameterisations for the learned noise distribution? In particular, I would be interested in whether the Gaussian parameterisation is sufficiently expressive, or whether that is in fact something of a limiting factor in the current procedure (but needed to ensure tractability).

On a related note, I liked the assessment of the state-dependent vs state-independent learned mean function. I wondered if the authors noticed any qualitative differences in the generations between these two approaches. In particular, was much variation was there in the generations between the two methods? I ask because without the state-dependence, the noise would have a similar correction applied each time, and this might results in images with similar structures.

Have the authors tested with gradient-information too? It seems that the regularisation method would also work with gradient-based optimisation, which for example would be available in the image experiments.

4 comments
AuthorsOpenReview ↗

Response to Reviewer JWzh (1)

We thank the reviewer for the constructive feedback on regularization, noise optimization, and methodological flexibility. We provide additional experiments and clarifications on these points, including regularization sweeps and gradient-based optimization, further supporting LeNO's effectiveness and flexibility. Detailed responses are provided below.

W1 & Q1 : Concerns on regularization and reward gain trade-offs

To examine the reward–regularization trade-off, we swept four regularization weights per method, including HyperNoise's default (0.5), using identical training prompts and 512,000 reward evaluations per model. All models were evaluated with SDXL-Turbo on the same 100 prompt–seed pairs. The penalties are

\mathcal{P}_{\mathrm{HN}} = \frac{\lambda_{\mathrm{HN}}}{d}\\,\mathbb{E}\left[\lVert\delta_\phi(z,c)\rVert_2^2\right]$$ where $d$ is latent dimensionality, $\eta\mu_\theta$ is LeNO's action mean ($\eta=0.3$, $\sigma=0.2$), and $\delta_\phi$ is HyperNoise's additive residual. Since LeNO uses conditional-action KL regularization with normalized advantages, whereas HyperNoise uses L2 regularization with raw rewards, we compare measured noise displacement rather than regularization coefficients. **At comparable measured noise displacement** (noise MSE: 0.01220 vs. 0.01298), **LeNO achieves a larger mean ImageReward improvement** (+0.2234 vs. +0.0738), suggesting that its advantage cannot be explained solely by the magnitude of noise corrections. We note that noise MSE is not equivalent to marginal KL. ImageReward gains do not consistently transfer to PickScore, but none of the evaluated LeNO settings shows a statistically significant PickScore decrease. This is a limited held-out check, not proof against reward hacking. We will include the full sweep and trade-off plots in the revision. Table 7. Regularization weight and reward gain sweeping results | Method | Coefficient (LeNO: β; HyperNoise: λ_HN) | ΔImageReward (95% CI) | Noise MSE ↓ | ΔPickScore (95% CI) | |---|---:|---:|---:|---:| | LeNO | 0 | +0.2099 ± 0.0891 | 0.01527 | +0.0505 ± 0.0981 | | LeNO | 0.01 | +0.2234 ± 0.0840 | 0.01220 | -0.0496 ± 0.0921 | | LeNO | 0.1 | +0.1046 ± 0.0585 | 0.00266 | +0.0457 ± 0.0737 | | LeNO | 1 | +0.0179 ± 0.0168 | 0.00038 | -0.0037 ± 0.0324 | | HyperNoise | 0 | +0.1239 ± 0.0752 | 0.02132 | +0.0554 ± 0.0697 | | HyperNoise | 0.05 | +0.0738 ± 0.0603 | 0.01298 | -0.0938 ± 0.0634 | | HyperNoise | 0.5 (default) | +0.0896 ± 0.0711 | 0.01822 | -0.0924 ± 0.0802 | | HyperNoise | 5 | +0.0771 ± 0.0737 | 0.01035 | +0.0921 ± 0.0776 | --- **Q2 : Question on Noise Optimization vs. Generator Fine-Tuning** We agree that directly fine-tuning the generator may offer greater flexibility for reward alignment. However, LeNO aims to improve reward alignment while preserving the capabilities of the pretrained generator. In this sense, **LeNO can be interpreted as a form of input-side model adaptation, where a learnable noise transformation acts as an additional input layer preceding the frozen generator**. Thus, rather than being entirely separate from generator fine-tuning, LeNO can be viewed as a constrained form of model adaptation that modifies the input distribution while preserving the pretrained generator's parameters and capabilities. Moreover, we believe that noise optimization offers a practical alternative in deployment scenarios where modifying or retraining an already deployed generative model is undesirable or computationally expensive. In such cases, learning a lightweight noise correction while keeping the pretrained generator frozen can provide an efficient and modular approach to reward alignment.
AuthorsOpenReview ↗

Response to Reviewer JWzh (2)

Q3 : The choice of Gaussian parameterization

The Gaussian parameterization applies to the residual action conditioned on the initial noise and prompt, rather than to the learned noise marginal. Before projection, the latter is

qθ(z′∣c)=∫N(z′;,z+ημθ(z,c),,σ2I)p0(z),dz,q_\theta(z'\mid c)=\int \mathcal{N}\left(z';\\,z+\eta\mu_\theta(z,c),\\,\sigma^2I\right)p_0(z)\\,dz,

which need not be Gaussian because the conditional mean depends nonlinearly on zz. Thus, Gaussian conditional actions do not restrict the transported noise marginal to a Gaussian family. This distinction is also illustrated by our multimodal toy experiment (Fig. 2), where the learned transport redistributes probability mass toward distinct reward-favorable modes despite using Gaussian conditional actions.

We choose Gaussian actions for their tractable likelihoods and closed-form conditional KL regularization. Gaussianity is not required for the KL upper bound itself, which follows from the data-processing inequality provided that the reference transition preserves the noise prior. For example, Gaussian-mixture actions may also admit a tractable upper bound through convexity of KL. Nevertheless, we acknowledge that the fixed isotropic Gaussian policy could potentially limit exploration in complex reward landscapes, although we have not observed clear evidence of such limitations in our experiments.


Q4 : Gradient-informed training of LeNO

We thank the reviewer for this insightful suggestion. Our framework is not restricted to gradient-free optimization: when reward gradients are available, the same stochastic residual transport and conditional-action KL regularization can be retained, while directly optimizing the expected reward through backpropagation instead of Local GRPO. This yields an objective related to HyperNoise, although the transport and parameterization differ.

To demonstrate this flexibility, we trained a gradient-based LeNO variant with ImageReward, retaining stochastic actions (η=0.3\eta=0.3, σ=0.2\sigma=0.2) and KL regularization (β=0.01\beta=0.01). The latest saved checkpoint available when this evaluation started had completed 512,000 training reward evaluations. On the same 100 benchmark prompt–seed pairs, it improves ImageReward with both mean-only and stochastic inference:

Table 8. Evalution of gradient-base LeNO training with ImageReward

MethodImageReward ↑ΔImageReward (95% CI) ↑
Original Noise0.7502--
LeNO (Local GRPO)0.9736+0.2234 ± 0.0840
Gradient-based LeNO / Mean-only0.8566+0.1064 ± 0.0532
Gradient-based LeNO / Stochastic0.8466+0.0963 ± 0.0449

Local GRPO's larger observed gain may reflect its use of 32 candidate actions per anchor and within-group advantage normalization, which learns from relative reward differences and reduces sensitivity to reward scale across anchors. Direct backpropagation instead uses the local reward derivative at a sampled action. This is a possible explanation, not an established cause: training budgets, optimizers, and evaluation protocols also differ, and a controlled ablation would be needed to isolate their effects.

Our contribution is a framework for amortized noise optimization that supports both gradient-based and gradient-free reward alignment, rather than a claim that gradient-free optimization is preferable when reward gradients are available. In particular, the gradient-free formulation enables alignment with non-differentiable black-box rewards, as demonstrated in our protein generation experiments.

Reviewer_JWzhOpenReview ↗

Response to rebuttal

I thank the authors for their detailed response, and my questions have been satisfactorily addressed. I have also read the other reviews, and will maintain my positive assessment of this work.

We thank the reviewer for their constructive feedback and are pleased that our rebuttal has addressed their concerns.

Comments on the paper

6 comments
Area_Chair_XQ5NOpenReview ↗

Emergency Review of Submission9479

Summary

The paper amortizes reward-guided initial-noise optimization for frozen few-step generators. A network μθ(z,c)\mu_\theta(z,c) predicts a residual mean. The transport samples a=ημθ+σϵa=\eta\mu_\theta+\sigma\epsilon and sets z′=NormProj(z+a;∥z∥)z'=\mathrm{NormProj}(z+a;\|z\|). Proposition 1 bounds KL(qθ∥p0)\mathrm{KL}(q_\theta\|p_0) by B=η22σ2E∥μθ∥2B=\frac{\eta^2}{2\sigma^2}\mathbb E\|\mu_\theta\|^2. Training uses "Local GRPO" with the penalty βB\beta B. Experiments cover a 2D toy, SDXL-Turbo, SANA-Sprint, one-step protein backbones, and DF-GAN.

Strengths

  1. Amortized noise correction from scalar, non-differentiable rewards is relevant for one-step generators and expensive rewards.
  2. Proposition 1 is simple and correct.
  3. The protein results are strong. LeNO has higher designability than per-sample ZeNO with no test-time reward queries, although ZeNO keeps a lower scRMSD on RMF-L.

Weaknesses

W1. Loss of diversity is not ruled out. The transport can map many anchors near one direction. A finite BB excludes collapse to a point, but not concentration of the corrected initial noise z′z'. To see this, let SS be any region that contains all the mass of qθq_\theta, i.e., qθ(S)=1q_\theta(S)=1, and consider the indicator z↦1[z∈S]z\mapsto\mathbb 1[z\in S]. Under qθq_\theta, this indicator follows Bern(1)\mathrm{Bern}(1); under p0p_0, it follows Bern(p0(S))\mathrm{Bern}(p_0(S)). By the data-processing inequality, the KL between these two Bernoulli distributions is at most KL(qθ∥p0)\mathrm{KL}(q_\theta\|p_0). This Bernoulli KL equals log⁡(1/p0(S))\log(1/p_0(S)). With Proposition 1, we get log⁡(1/p0(S))≤KL(qθ∥p0)≤B\log(1/p_0(S))\le\mathrm{KL}(q_\theta\|p_0)\le B, i.e., p0(S)≥e−Bp_0(S)\ge e^{-B}. Thus the bound only forbids concentration on regions of prior mass below e−Be^{-B}. This threshold is tight: for any SS with p0(S)≥e−Bp_0(S)\ge e^{-B}, the prior conditioned on SS has KL exactly log⁡(1/p0(S))≤B\log(1/p_0(S))\le B, so Proposition 1 cannot exclude it. (This does not show that the transport attains such a distribution, since BB only upper-bounds the KL.) The same argument applies to the generated images: by data processing through the generator, the KL between the distributions of generated images with and without correction is also at most BB. Whether such concentration reduces image diversity cannot be read from the bound, so the claim rests on the empirical evidence.

W2. Related work and baselines are incomplete. The paper misses some closely related prior works, e.g., Noise PPO (Miao et al., arXiv:2506.12036), which trains a prompt-conditioned initial-noise policy with PPO for a frozen diffusion model.

Questions

  1. For the image models: which timestep does μθ\mu_\theta receive, is the output raw or a difference, is μ=0\mu=0 at initialization, and how many parameters are trainable? What is the SDXL-Turbo time and memory cost? What computes μθ\mu_\theta for DF-GAN?
  2. How many steps was each image model trained for, and which checkpoint does Table 1 use? Please report diversity at that checkpoint, a distribution-level metric, and diversity for SANA-Sprint and across σ\sigma and β\beta (including β=0\beta=0).
  3. How does LeNO compare with Noise PPO?

rating: 2

confidence: 2

AuthorsOpenReview ↗

Response to Area Chair XQ5N (1)

W1 : Loss of diversity

We thank the reviewer for this constructive comments. We agree that a finite KL upper bound does not guarantee diversity preservation or rule out concentration on subsets of the original distribution. Our theoretical result is intended to bound the distributional deviation from the prior, rather than to provide a formal guarantee against diversity collapse. We will clarify this distinction in the revised manuscript.

Our diversity-preservation claims are instead supported by empirical evidence. Beyond the results reported in Appendix E.1 (Fig. 7 and Table 7), we additionally evaluated diversity across different regularization strengths β\beta (including β\beta=0), using the same 100 prompts × 8 fixed seeds. The values below report diversity retention relative to the unmodified SDXL-Turbo baseline (100%), with paired prompt-bootstrap 95% confidence intervals.

Table 1. Diversity retention across reward objectives and KL regularization strengths.

Checkpoint / AblationTraining StepLPIPS Retention (%)DINO Retention (%)
Aesthetic1,00098.9 [98.2, 99.6]99.6 [95.9, 103.7]
PickScore5,000100.2 [99.1, 101.3]98.9 [93.3, 105.0]
ImageReward2,500106.2 [105.0, 107.4]108.0 [101.4, 115.4]
ImageReward, β\beta=01,000108.0 [106.9, 109.1]110.1 [103.8, 116.9]
ImageReward, β\beta=0.011,000107.8 [106.8, 108.9]108.3 [102.2, 114.5]
ImageReward, β\beta=0.11,000101.9 [101.3, 102.6]103.2 [99.6, 106.9]
ImageReward, β\beta=11,000100.0 [99.9, 100.1]100.0 [99.1, 101.0]

The results show no substantial diversity collapse across the evaluated settings, including without KL regularization. At the evaluated checkpoints, PickScore retains 100.2% LPIPS and 98.9% DINO diversity, while ImageReward retains 106.2% and 108.0%, respectively. Aesthetic exhibits only a modest decrease (98.9% LPIPS and 99.6% DINO).

We acknowledge that these measurements characterize within-prompt diversity rather than full distributional fidelity or mode coverage, and therefore do not establish a general guarantee against concentration. We will revise the manuscript to clearly distinguish the theoretical role of KL regularization from the empirical evidence for diversity preservation.

AuthorsOpenReview ↗

Response to Area Chair XQ5N (2)

W2 & Q3 : Comparison with NoisePPO

We thank the reviewer for pointing out this closely related work. We agree that Noise PPO shares our motivation of improving reward alignment by learning initial-noise distributions while keeping the pretrained generator frozen. We will include and discuss this work in the revised manuscript.

However, the two approaches differ in their target, policy parameterization and optimization strategy. Noise PPO directly learns a prompt-conditioned Gaussian initial-noise distribution, πθ(z∣c)=N(μθ(c),Σθ(c))\pi_\theta(z\mid c)=\mathcal{N}(\mu_\theta(c),\Sigma_\theta(c)), using PPO with a learned value function. In contrast, LeNO learns an anchor- and prompt-conditioned residual transport, where Gaussian actions are sampled around μθ(z,c)\mu_\theta(z,c) and optimized through Local GRPO using group-relative reward comparisons within each anchor. Consequently, unlike Noise PPO's explicitly Gaussian noise marginal, LeNO induces an implicit and generally non-Gaussian corrected-noise distribution. Furthermore, our KL regularization upper-bounds the divergence of this implicit marginal from the original prior through the conditional action KL.

Since an official implementation was not available, we independently reimplemented Noise PPO and compared it with LeNO using SDXL-Turbo and PickScore optimization, with ImageReward as a held-out reward. Both checkpoints were trained for 1,500 iterations (768,000 generated images/reward evaluations) and evaluated on the same 100 prompt–seed pairs, using one-step generation and stochastic inference. Our Noise PPO variant uses KL weight 1 and entropy weight 0.1. It also uses PickScore alone rather than the paper's composite reward. Thus, this is an adapted baseline comparison, not an exact reproduction of the published setting.

Table 2. Comparison with NoisePPO

MethodPickScore ↑ImageReward ↑ (held-out)
SDXL-Turbo21.78490.7465
Noise PPO (our reimplementation)21.80710.7621
LeNO22.10910.9125

Relative to the shared baseline, PickScore gains are +0.0223 for Noise PPO and +0.3242 for LeNO; held-out ImageReward gains are +0.0156 and +0.1660, respectively. Training budgets are matched by generated-image count.

Nonetheless, these results, together with the methodological differences above, distinguish LeNO from directly learning a prompt-conditioned Gaussian noise distribution. In particular, LeNO focuses on amortizing sample-wise reward-improving corrections through local, anchor-conditioned learning, rather than directly optimizing a global noise policy for each prompt. This design is further supported by our state-dependency ablation (Fig. 4), where removing anchor conditioning leads to substantially weaker reward improvement.


Q1 : Implementation Details

  • Timestep. The residual backbone receives a fixed high-noise timestep: t=999t=999 for SDXL-Turbo and t=1t=1 in SANA-Sprint's SCM convention.

  • Output and initialization. μθ(z,c)\mu_\theta(z,c) is computed as fθ(z,t,c)−f0(z,t,c)f_\theta(z,t,c)-f_0(z,t,c) where f0f_0 is frozen, not a raw noise prediction. Zero-initialized LoRA output matrices give μθ=0\mu_\theta=0 initially. The action mean is ημθ\eta\mu_\theta and stochastic training additionally samples σϵ\sigma\epsilon.

  • Trainable parameters. The counts below exclude frozen generator/text-encoder weights.

    Table 3. The number of trainable parameters

    ModelTrainable parameters
    SDXL-Turbo (rank-16 LoRA)49.41M
    SANA-Sprint (rank-16 LoRA)8.26M
    Protein-RMF (RMF-M, rank-32 LoRA)7.08M
    Protein-RMF (RMF-L, rank-32 LoRA)17.2M
    DF-GAN (conditional residual MLP)0.195M
  • SDXL-Turbo cost. For the previously benchmarked checkpoint, one-step 512×512512\times512 generation takes 0.304 s/image on one RTX 4090, versus 0.130 s/image without LeNO. ImageReward training is estimated at 32.32 s/iteration on eight RTX 4090s, assuming ideal linear scaling from our two-GPU measurements, with 512 generated/reward-scored images and four policy optimization epochs per iteration. Peak PyTorch-allocated memory in the measured two-GPU run was 18.03 GiB per GPU. These costs refer to the previously benchmarked checkpoint, not a separate benchmark of the LoRA-only configuration.

  • DF-GAN implementation. A separate conditional residual MLP maps the 100-dimensional noise and 256-dimensional frozen sentence embedding to a noise correction, using hidden width 64 and four conditional residual blocks. The formulation becomes μθ=NoiseMapper(z,c)−z\mu_\theta=\mathrm{NoiseMapper}(z,c)-z and its output projection is zero-initialized. The generator remains frozen and no diffusion timestep is used.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.