acceptodds
Withdrawn from ICLR 2027

Noise in Diffusion Models Is a Learnable Input

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model training consumes concrete random inputs that enter the realized loss and optimization. Because training aims to reduce this loss, models can learn to exploit accessible structure in these inputs, making realized randomness itself learnable content. We establish this learning mechanism in diffusion models, where clean data and target noise jointly form the noisy input. Prediction gains can arise from clean-data regularities, noise structure, and their interaction, so lower loss need not indicate better learning of clean-data regularities. We test this mechanism with controlled, reproducible pseudorandom number generator (PRNG) streams on MNIST and CIFAR-10. Random-role ablations localize the dominant effect to diffusion noise. Without reusable clean-image structure, structured-noise training can lower loss below the IID reference, while IID testing reverses this advantage, demonstrating learned noise dependence. Value-preserving shuffling largely removes these gains, identifying ordering as a major source of learnable structure. Matched-source initialization improves representative generations, supporting learned noise as a generation cue. Prior results on noise–data assignment, noise-based backdoors, and structured video noise provide independent evidence of learned use of noise structure and noise–data relationships, including in large generative models. These studies also demonstrate that noise design can exploit this learnability to achieve specific goals or improve model performance. These findings motivate testing whether existing large generative models learn structure in industrial-grade PRNG outputs that affects generation quality. Our view establishes noise as a learnable and therefore designable input dimension for guiding learning and generation.

Primary area:
generative models
Keywords:
diffusion models, noise learnability, structured noise, noise-data interactions

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_b7RpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether diffusion models can learn and exploit statistical structure in the noise used during training. The authors show that structured, non-IID noise can reduce denoising loss and influence generation, indicating that diffusion models may rely not only on data structure but also on regularities in the noise itself. Based on these observations, the paper argues that diffusion noise can itself provide learnable structure and may therefore serve as a controllable component of the generative process.

Critical strengths

None

Critical weaknesses

The most fundamental problem with this paper is that its main conclusion is not meaningful for generative modeling. The paper shows that, once the noise contains exploitable structure, the model can use that structure to predict the noise more accurately, but this is essentially expected and does not address the actual goal of a generative model, which is to learn the data distribution rather than the noise distribution. In standard diffusion models, noise prediction is meaningful only because, under the Gaussian corruption formulation, predicting ϵ\epsilon is a particular parameterization of learning the score function of the corrupted data distribution and is therefore tied to modeling pdatap_{\mathrm{data}}. Once additional predictable structure is introduced into the noise, the model can reduce the prediction loss by modeling the corruption process itself rather than by learning more about the data distribution, while learning such noise structure is itself irrelevant to the generative objective. In the extreme case of deterministic noise ϵ=ϵ0\epsilon=\epsilon_0, the model could achieve zero prediction loss simply by always outputting ϵ0\epsilon_0, while learning nothing about pdatap_{\mathrm{data}}. This example highlights that improved predictability of the noise can be completely decoupled from improved modeling of the data distribution. Structured noise is simply a less extreme version of the same issue, so demonstrating that such noise is "learnable" does not constitute a meaningful result for diffusion-based generative modeling.

Questions for authors
  1. The paper shows that structured noise can make the noise-prediction problem easier, but it is unclear how this relates to the actual objective of generative modeling. Can the authors provide a principled argument, beyond improved noise-prediction loss, showing that exploiting noise structure leads to better modeling of pdatap_{\mathrm{data}} rather than merely better modeling of the corruption process?

  2. Once the noise distribution is changed from IID Gaussian noise to a structured non-IID distribution, the forward corruption process is also changed. What is the theoretically corresponding reverse process in this setting? In particular, is the standard DDIM sampling rule still theoretically justified?

Reviewer_uaxLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether diffusion models can learn and exploit statistical structures present in the noise used during training. The authors argue that noise should not be regarded merely as a source of randomness, but also as a learnable input that can influence optimization and generation.

To support this claim, the paper conducts experiments using several pseudorandom number generators (PRNGs) on MNIST and CIFAR-10. The authors show that structured noise can reduce noise-prediction loss even when meaningful clean-image structure is removed. They further demonstrate that shuffling noise sequences largely eliminates these improvements, suggesting that the model exploits statistical dependencies associated with noise ordering.

The paper also examines train–test mismatches across noise sources and reports improvements in representative generated images when using matched-source noise during sampling.

While the experimental observations are interesting, I have substantial concerns regarding the novelty of the proposed learning mechanism, the interpretation of PRNG learnability, and the practical significance of the findings.

Critical strengths

None

Critical weaknesses
  1. Even in an ideal IID-clean setting, any noise you generate is produced by a PRNG, meaning it can also be considered structured noise. Therefore, all the noise you generate consists of different types of structured noise, and I do not clearly understand what you are trying to demonstrate. Since all generated noise is pseudorandom and influenced by the seed and PRNG, truly ideal IID noise does not exist.

  2. If the noise is structured, is it possible to predict that noise? Please show the experimental results.

  3. When the seed changes, the PRNG generates a different sequence. In that case, does changing the seed degrade generation performance? Or does the model still generate images well because the underlying noise structure remains the same?

  4. Noise is an input to the model, so it seems natural that the model can learn from it. Furthermore, is it possible to identify the randomness generated by a PRNG? Have you examined the length of the sequence generated by the PRNG, from its initial value to its final value? Does this sequence repeat multiple times?

Questions for authors

Please refer to the Critical Weaknesses section.

Reviewer_Bb3BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether diffusion models learn exploitable structure in their noise inputs. Using controlled pseudorandom streams on MNIST and CIFAR-10, the authors show that source-dependent reductions in noise-prediction loss persist when clean images are replaced with fresh tensors. Cross-source evaluation reveals learned dependence on the training noise source, while value-preserving shuffling removes much of the prediction advantage. The paper presents qualitative generation experiments and interprets several existing methods through the lens of noise learnability.

Critical strengths
  1. The IID-clean probe is a useful experimental design. Removing reusable image structure makes the evidence for noise-dependent prediction substantially more convincing than comparisons on natural images alone.

  2. The controlled experiments provide evidence that diffusion models exploit dependencies in noise tensors. The combination of random-role ablations, cross-source evaluation, and value-preserving shuffling provides complementary evidence.

Critical weaknesses
  1. In Sections 2.1 and 4.4, authors emphasize concrete realizations and finite trajectories, whereas most experiments establish adaptation to a structured noise distribution. I think they are different claims. The authors should distinguish distribution-level dependence, source-level generalization, and trajectory-specific specialization, and state which experiments establish each.

  2. The experiments are limited on MNIST and CIFAR-10. Although these controlled settings support the proposed mechanism, it remains unclear how the findings generalize to other datasets and larger models. The authors need to give more details.

Questions for authors
  1. How do the authors distinguish trajectory-specific learning from differences in the statistical properties of the training and test streams? Could the SS8 train–test divergence arise from nonstationarity or finite-precision effects?

  2. Can the authors provide an additional experiment on other datasets to assess whether the observed noise dependence extends beyond MNIST and CIFAR-10?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.