acceptodds
Withdrawn from ICLR 2027

NoiseAR: AutoRegressing Initial Noise Prior for Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion models have emerged as powerful generative frameworks, creating data samples by progressively denoising an initial random state. Traditionally, this initial state is sampled from a simple, fixed distribution like an isotropic Gaussian, inherently lacking structure and a direct mechanism for external control. While recent efforts have explored ways to introduce controllability into the diffusion process, particularly at the initialization stage, they often rely on deterministic or heuristic approaches. These methods can be suboptimal, lack expressiveness, and are difficult to scale or integrate into more sophisticated optimization frameworks. In this paper, we introduce NoiseAR, a novel method for AutoRegressive Initial Noise Prior for Diffusion Models. Instead of a static, unstructured source, NoiseAR learns to generate a dynamic and controllable prior distribution for the initial noise. We formulate the generation of the initial noise prior's parameters as an autoregressive probabilistic modeling task over spatial patches. This approach enables NoiseAR to capture complex spatial dependencies and introduce learned structure into the initial state. Crucially, NoiseAR is designed to be conditional, allowing text prompts to directly influence the learned prior, thereby achieving fine-grained control over the diffusion initialization. Our experiments demonstrate that NoiseAR can generate initial noise priors that lead to improved sample quality and enhanced consistency with conditional inputs, offering a powerful, learned alternative to traditional random initialization. A key advantage of NoiseAR is its probabilistic formulation, which naturally supports seamless integration into probabilistic frameworks like Markov Decision Processes and Reinforcement Learning. Cross-model evaluation supports generalization beyond the generators used to construct training noise, while preference and reward-based fine-tuning further improve the learned prior. Furthermore, NoiseAR acts as a lightweight, plug-and-play module, requiring minimal additional computational overhead during inference, making it easy to integrate into existing diffusion pipelines.

Primary area:
generative models
Keywords:
Noise Prior, Generative Models, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_em9GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NoiseAR, a method that learns a prompt-conditioned initial noise prior for diffusion models. Instead of sampling noise from a standard Gaussian distribution, NoiseAR uses an autoregressive model to generate noise patches sequentially. The model predicts the mean and variance of each patch based on previous patches and text conditions. It is trained using noise samples obtained through a Weak-to-Strong inversion procedure. The probabilistic formulation also allows further optimization through DPO and GRPO using image-level feedback. Experiments across several diffusion models and benchmarks demonstrate improvements in image quality and text-image alignment.

Critical strengths
  • Interesting research direction. The paper explores learning a conditional initial noise distribution rather than modifying the denoising process. This offers an alternative approach to improving controllable image generation without changing the pretrained diffusion model.

  • Flexible probabilistic formulation. NoiseAR uses autoregressive modeling to capture spatial dependencies between noise patches. Its explicit probability distribution also supports preference-based and reinforcement learning methods such as DPO and GRPO, making the framework flexible for further optimization.

  • Encouraging experimental results. The paper evaluates NoiseAR on multiple diffusion models and benchmarks, including SDXL, DreamShaper, Hunyuan-DiT, and Qwen-Image. The results show improvements over standard Gaussian initialization and Golden Noise on several image quality and alignment metrics. The additional cross-model experiments and reward-based fine-tuning results further demonstrate the potential of the approach.

Critical weaknesses
  • Insufficient ablation of the autoregressive design. The main contribution of the paper is learning a conditional initial noise prior using an autoregressive model. However, the ablation studies mainly focus on patch size and network depth. There is no direct comparison with simpler alternatives, such as independent patch-wise Gaussian modeling or non-autoregressive conditional models. Therefore, it remains unclear whether the performance improvements come from autoregressive modeling or simply from learning a prompt-conditioned noise distribution. This weakens the justification for the core architectural design.

  • Limited comparisons with existing initial noise optimization methods. The experiments mainly compare NoiseAR with standard Gaussian initialization and Golden Noise. However, several relevant methods, such as InitNO and ReNO, are discussed in the related work but not included in the experiments. A broader comparison would help establish the advantages of NoiseAR in generation quality, computational efficiency, and generalization. Without these comparisons, it is difficult to fully assess its effectiveness relative to existing approaches.

Questions for authors

How does NoiseAR maintain compatibility with the Gaussian prior assumed by pretrained diffusion models? Since NoiseAR learns a prompt-conditioned initial noise distribution, the generated noise may deviate from the standard Gaussian distribution used during diffusion model training. Could the authors analyze the statistical properties of the learned noise, including its mean, variance, and distributional differences from a standard Gaussian? It would also be helpful to clarify whether such deviations affect generation diversity, sample quality, or generalization across diffusion models.

Reviewer_6UCsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces NoiseAR, a novel framework that replaces the traditional, unstructured isotropic Gaussian initialization in diffusion models with a learned, prompt-conditioned autoregressive initial noise prior. Instead of treating the starting noise as a static and uncontrollable random state, NoiseAR factorizes the 3D noise tensor into spatial patches and models their conditional probability distribution autoregressively using a Transformer Decoder. This allows the generation process to be directly influenced by text prompts from its absolute inception. Furthermore, because NoiseAR maintains an explicit probabilistic likelihood, it can be seamlessly integrated into reinforcement learning frameworks like Markov Decision Processes (MDPs) for reward-based adaptation and preference optimization (such as DPO and GRPO).

Critical strengths

Unlike existing heuristic or deterministic initialization methods that rely on single-point predictions or fixed mappings, NoiseAR formulates the initial noise prior as a conditional probability distribution over spatial patches using a Transformer Decoder. This is highly consequential because it preserves generation diversity, captures complex spatial dependencies, and provides an explicit likelihood. As a result, it bridges the gap between diffusion initialization and probabilistic frameworks like Markov Decision Processes (MDPs).

Because NoiseAR explicitly defines a tractable likelihood over the initial noise, it naturally supports policy-style optimization using image-level human preferences and rewards without needing to differentiate through the heavy downstream generator. Combined with its plug-and-play nature—introducing negligible computational overhead (adding only ~0.2% time cost)—this demonstrates high practical utility, making it exceptionally easy to integrate into existing diffusion pipelines.

Critical weaknesses

The paper's core premise—that the quality of initial noise matters and that manipulating or predicting initial noise can improve text-to-image generation—is not entirely new. Prior works such as InitNO, ReNO, and specifically Golden Noise have already explored direct text-to-noise prediction or noise optimization. Although NoiseAR introduces an autoregressive probabilistic formulation to distinguish itself from deterministic single-point predictors like Golden Noise, the conceptual leap of conditioning initial noise on text prompts is already established in the literature. Consequently, the incremental conceptual novelty over prior direct noise-prediction frameworks may be viewed as limited, making it harder to substantiate a major methodological breakthrough.

Questions for authors

While NoiseAR uses an autoregressive probabilistic formulation (NLL objective) compared to Golden Noise's deterministic MLP prediction, could the authors clarify how much of the performance gain is specifically attributable to the autoregressive patch-level distribution versus the Weak-to-Strong data construction pipeline? Specifically, how would Golden Noise perform if it were trained using the exact same NLL-based probabilistic patch formulation, or conversely, how would a non-autoregressive conditional Gaussian head perform under the same pipeline? Clarifying this distinction is crucial to substantiate the true methodological advantage of the AR prior.

The paper mentions that cold-start data is constructed using a Weak-to-Strong inversion pipeline on Pick-a-Pic prompts. To what extent does NoiseAR's learned prior inherit the specific biases, failure modes, or stylistic traits of the upstream generator used during data construction? Furthermore, when transferring the prior to Qwen-Image (where the 4-channel noise is replicated to 16 channels), did you observe any channel-mismatch artifacts or degradation in out-of-distribution text prompts not covered by the training distribution?

Reviewer_NoyROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a new method for controlling the generation of diffusion models that involves changing the prior distribution. The authors propose a small AR transformer architecture that takes as input both the initial noise zTz_T drawn from an isotropic Gaussian and the textual prompt, and outputs the prompt-conditioned prior zT^\hat{z_T}. They also show NoiseAR can be further fine-tuned using feedback from its own generated samples using RL methods such as DPO and GRPO. Their method outperforms important baselines on several metrics.

Critical strengths
  • The idea of learning the noise prior autoregressively is simple, yet interesting.
  • The method can be used plug-and-play and it's efficient at inference.
Critical weaknesses
  • The Related Work is missing from the main paper. Right now, the Method section is too verbose (i.e., about four pages long). Most of the things can be summarized in a few sentences such as explaining the key components of NoiseAR. I'd rather reduce the sections 2.2.1 - 2.2.4. to just a few bullet points, move most of the details in the Appendix, and then add a short version of the Related Work section in the main paper. For instance, section 2.2.1 is half-page long and doesn't describe anything (completely) new compared to a regular ViT - it could be summarized in 1-2 sentences that e.g., you add a learnable vector Start Token Embedding that "serves as an initial input to the model".
  • The Related Work seems to be very outdated (almost all papers have been published before 2025). I would also like to see in the paper a dedicated paragraph that discusses / enumerates the line of related prior work that does RL-optimized noise distributions. One paper that I find relevant is Noise Hypernetworks (NeurIPS 2025) [1].
  • Section 3 does not provide any experiments that make us understand the method better. It is very clear in Section 2 what you did, but I cannot see in the paper (particularly in Section 3) why you made certain design choices. The ablations on the core architecture are alright, but I believe they do not really answer any research question. For example:
    • is modeling AR-wise better than using e.g., a masked objective?
    • What about NLL vs. MSE using the same architecture?
    • What happens if we only learn the mean μ\mu and keep σ\sigma constant?
  • I see several "by default" mentions in the paper (e.g., "use one layer in each component by default", "raster scan order by default", "sinusoidal functions by default")... a natural question is whether you also tested other design choices. If so, can they be added in the paper? If not, could you remove this formulation?

[1] Eyring, L., Karthik, S., Dosovitskiy, A., Ruiz, N., & Akata, Z. (2026). Noise hypernetworks: Amortizing test-time compute in diffusion models. Advances in Neural Information Processing Systems, 38, 95429-95462.

Questions for authors

See weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.