acceptodds
Under review as a conference paper at ICLR 2027

Noise-Level Adjacency in Diffusion Training

Abstract

Diffusion models train one network across all noise levels, although their training objective is pointwise in the noise level. If training on each pair of adjacent levels were an independent problem, separate networks with the same total parameter count would be expected to match one shared network. But in practice, on images, sharing weights across levels gives better results, which hints at an implicit dependence on their order. An order fixes both which levels are adjacent and the direction in which they run. In this paper, we show that, of the two, the benefit of sharing rests on adjacency in the noise-level embedding. The fitting error of a shared network is bounded below by the distances between neighbouring posterior means, minus the change the network can make between neighbouring embeddings. We first distinguish adjacency from direction: on a Gaussian mixture and CIFAR-10, reversing the embeddings keeps every neighbour and leaves the fitting error unchanged, whereas shuffling them raises it. Second, we rule out sharing between distant levels as the source of the benefit: separate networks on contiguous ranges match the shared network at its width, and with the parameters fixed, breaking adjacency at the same places costs less than 40% of a full shuffle. Dividing the levels at equal shares of these distances lowers the error of all but the smallest networks. We therefore show that the order of the noise levels enters a diffusion model through adjacency, with its direction being a design choice.

Primary area:
generative models
Keywords:
Diffusion, Generative AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sxbfOpenReview โ†—
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates the reason why training one network acroll all noise levels in diffusion models performs better than training separate networks for each noise level. It is shown that the objectives are independent for each noise level k, and how the noise level embedding influences the shared network error. Numerical experiments illustrate that splitting the noise levels across separate networks matches or improves performance if the width is the same for each network, but deminishes performance if the total number of parameters is kept fixed, with better performance for contiguous ranges of noise levels. By experimentally analyzing the effects of reversing or shuffling the noise levels in the embedding, the authors conclude that adjacency and not direction is the key to increased performance.

Critical strengths

The experiments are rather extensive and test various setups.

Critical weaknesses

Three main problems with the paper: (1) previous studies already revealed that adjacency plays a key role in training a shared network for diffusion models, (2) the theoretical results are unsurprising and can be derived in a straightforward manner, and (3) the paper is not well written.

The closest prior work, Go et al. (2023) and Park et al. (2024), is not discussed in the main paper, and the results appear to overlap substantially with it. In particular, the papers already demonstrate that adjacency of noise levels plays a key role in training diffusion models. Go et al. observe that the task affinity between denoising tasks is high for adjacent noise levels and descreases smoothly as the difference increases. Park et al. propose a method that uses this task affinity by activating similar channels for tasks at adjacent timesteps and show that it consistently boosts performance. Against this background, the additional insights from the present paper seem minimal, diminishing its novelty and significance.

Theorem 1 is almost immediate and Lemma 1 follows directly from the assumptions.

Some key mathematical quantities in the paper are not properly defined (e.g. ๐น_๐‘˜, ๐‘“, product of argmin). The paper is partly hard to read since Tables and Figures show results early on that are explained much later in the paper (e.g. Table 1 on page 4 includes results for โ€œevery K-thโ€ which is not explained until Section 5 on page 8). The explanation with reference to Figure 2c on page 8 lines 424-426 is unclear. Moreover, the paper moves all related work to the appendix, which prevents readers from judging the contribution in context.

Questions for authors

What are the additional insights from the present paper compared to Go et al. (2023), and why are they significant?

Is there a reason/intuition why the graphs in Figure 6 have this shape?

When using several seeds in the experiments, it seams reasonable to also show the variance (not just the mean).

4 comments

Thank you for acknowledging that our experiments are extensive and test various setups. Table numbers refer to the revised version.

W1 and Q1: Go et al. (2023) and Park et al. (2024) build methods on the observation that nearby timesteps are similar. We ask a different question: the training objective does not contain an order of the noise levels (Theorem 1), so where can an order enter a model, and which part of it does the network use? Their observation cannot answer this question, because it is made under one fixed embedding. Under this embedding, noise levels with similar denoising tasks are also adjacent in the embedding, and the embedding always runs in the same direction, so the effects of task similarity, adjacency and direction cannot be separated. For this reason, we only change the embedding and keep the tasks fixed. Since the tasks are the same, any change in the error must come from the embedding, and we find that it depends on which levels are adjacent and not on their direction. This result is thus novel.

It is significant because many previous work showed that much of the physical description of diffusion models is a design choice, but the order of the noise levels has remained fixed in every formulation. We show that only its adjacency is needed, so its direction is also a design choice. We now discuss both works in Section 6.

W2: Our proofs are short, but this does not mean that the question itself is simple. Diffusion models are formulated along a time axis, yet Theorem 1 shows that their objective treats the noise levels as an unordered family. So if the order matters, it can only enter through the network, which motivates our question. It also means that relabelling the embeddings keeps the objective fixed (Corollary 1), so a change in the error after relabelling can only come from the network.

Lemma 1 uses one triangle inequality. Its contribution lies in the choice of quantities: it expresses the fitting error in terms of the distances between posterior means. Because these distances are determined by the data before any training, the bound predicts which embeddings fit worse.

W3: We now define the section f(ยท, k) and the posterior mean F_k in Definition 1 and write out the product of argmin in Theorem 1. We also define the interleaved (every K-th) division in Section 3 before Table 1, rewrite the passage on Figure 2c, and move the closest related work to the main text (Section 6).

Q2: The shape comes from two effects. First, a network that covers a narrower range at the same width fits its levels better, so the curves lie below zero over most of the grid. Second, a transition next to a boundary gains less, because its network contains its neighbours on one side only (Table 2). The grey line separates the two effects: networks that each cover every 4th transition get no narrowing, and they stay close to one network everywhere. This is why the gain is smallest next to the boundaries and largest inside the ranges.

Q3: We add the standard deviation beside every mean over seeds in Tables 4, 5 and 6.

If these address your concerns, we would be grateful if you could raise your score.

Thank you for the response and the revised manuscript. The changes address part of my concerns. However, two concerns remain.

The first is the theoretical contribution, which I still consider minor. The rebuttal explains the motivation for the questions, but this does not change that Theorem 1 and Lemma 1 follow almost immediately from the setup, so the value of the paper lies in the experiments rather than the theory. You state that Lemma 1 predicts which embeddings fit worse, but the paper does not show that the bound tracks the empirical ranking of embeddings (e.g., contiguous vs. interleaved vs. shuffled). Such a comparison would make the theory more meaningful. Going further and quantifying in theory how the different setups affect the error (e.g., contiguous vs. interleaved splits, or the number of networks at a fixed parameter count) would be a valuable addition, though I realize this may be beyond the scope of the paper. I would also encourage stating more carefully that the evidence for direction being a design choice is limited to the tested setups.

The second concern is Figure 6. Your explanation accounts for the shape of the gain relative to a single network (narrower ranges fit better, and boundary transitions have neighbours on one side only). My question was about the loss pattern described in the manuscript, namely why the loss increases toward the clean end of the first three ranges and peaks at the noisy end of the last. The response does not explain this, and I would appreciate either an explanation or a clear statement that it is an empirical observation you do not yet understand.

Thank you for the constructive follow-up.

For the first concern, we agree that the main value of the paper lies in the experiments, but the experiments depend on the theory. Theorem 1 is what allows us to change the embedding while the objective stays the same, so that a change in the error can only come from the network. Lemma 1 gives a bound that is computed from the data before any training, and it should predict which embeddings fit worse. To test this, we ran the comparison you suggest. We rank the embeddings (11 on the mixture, the 6 of Table 3 on CIFAR-10) by the bound and compare this ranking with the ranking by the measured error:

Spearman correlation between the two rankings
mixture, width 2560.86
mixture, width 5120.86
mixture, width 10240.27
CIFAR-100.99

The two rankings agree well on the mixture at widths 256 and 512 and on CIFAR-10. At width 1024 the correlation is low because all embeddings except the full shuffle have almost the same error, with differences comparable to the variation across seeds. For the divisions you mention, the bound is lowest for contiguous ranges, does not increase for interleaved ones, and is highest for a shuffle, and the measured errors follow the same order. Thus we can conclude that the bound does track the empirical ranking. Quantifying these effects in theory at a fixed parameter count is beyond the scope of this paper, as you note.

We agree that the evidence for direction being a design choice is limited to the settings we test, and we now state this as a limitation in the conclusion.

For the second concern, the pattern comes from two effects. First, a transition next to a boundary has neighbours only on one side, so its loss relative to a single network is higher (Table 2). Second, in the first and the last range, the gain from separate networks is larger towards the end of the grid, where EDM's training distribution has few samples and a single network is least trained. In the first range both effects point to the clean end, and in the last range both point to the noisy end, which is why the loss is highest there. The two middle ranges have a boundary at both ends, but their loss rises mainly towards the clean end. This asymmetry is the only part that we cannot yet explain.

We hope this addresses your concerns.

Thanks for the response; following up on the discussion, I would raise my score to 2.

Reviewer_FopdOpenReview โ†—
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors consider the ordering of noise levels in diffusion model training, and investigate whether using a single unified network is better than a collection of separate networks, and if so what the effects are of noise adjacency and order. The authors start with a theoretical formulation, then continue to perform various experiments on image datasets, as well as a gaussian mixture. They study the effects of randomly shuffling noise levels, using contiguous noise blocks, as well as strided noise blocks.

Critical strengths
  • I appreciate the experimental approach to this problem, and the paper does have a significant amount of results in it
Critical weaknesses
  • the presentation is quite unclear in places, with lots of refering to material ahead in the paper (e.g. in the introduction), or refering to tables out of order (section 5), as well as relying overly on putting results in the appendix this makes reading difficult. Moreover, in my opinion a paper should be readable without having to read to the appendix, which in this case it is not. In my opinion, the paper needs to be carefully organized, and the authors should make decisions about what they want to present before it is ready for publication.
  • The problem the authors discuss seems to boil down to studying what the effect is of applying permutations to the noise levels using various embedding strategies, and whether it helps to split neural networks across noise levels. It seems to me that this is highly dependent on how the exact architecture of the neural net, and how noise levels are embedded, and it is not clear to me what it is about the diffusion process itself that matters for the results here. It seems like the problem reduces to how easy it is to learn particular input mappings, rather than an inherent property of diffusion models.
Questions for authors
  • Are all the tables FID? It is not specified after the first table
  • I am curious if the authors have thought about how this maps to models with continuous noise representations? In those cases, "shuffle" would for example no longer be a valid experiment. This seems like maybe an interesting question to answer, since in the continuous case, the model is defined over all reals, and can actually be tested on an untrained example and give OK results. This implies some kind of inductive bias for which I think both adjaceny and order would be important.
1 comment

Thank you for acknowledging our experimental approach and the amount of our results. Table numbers refer to the revised version.

W1: We listed the references in the introduction to help readers locate the corresponding results and claims. Since the reviewer points out that they made the paper harder to read, we remove them in the revised version. We also move the key related work and the key results of the previous appendix (retraining and replacing networks, Appendix E of the submitted version) to the main text, so that the paper can be read without the appendix. Section 5 now contains its own tables in the order it cites them.

Q1: No. We report FID on 50k samples in Tables 1 and 7. In the other CIFAR-10 tables of the main text we report the held-out denoising loss, on 10k held-out pairs per transition with the training weight, and in the mixture tables the excess error, the squared distance to the posterior mean (Section 2). We now name the metric of each table in its header as well as in its caption.

W2: It is valid to see this as an input mapping problem, but the difficulty of the mapping is determined by the diffusion process. The network must fit the posterior mean at each noise level, and the data and the forward process determine these posterior means before any architecture or embedding is chosen. A mapping is hard when the posterior means at neighbouring embeddings are far apart, so how hard it is depends on the diffusion process. The architecture only determines how much a network can change between neighbouring embeddings. This is why we observe the same result on an MLP, on a U-Net and on a transformer (Figure 4, Table 3).

Q2: Our CIFAR-10 networks are already continuous-noise models, since we train them on ฯƒ drawn continuously from EDMโ€™s distribution with EDMโ€™s sinusoidal embedding of log ฯƒ. A shuffle is still a valid experiment because we can divide the range of ฯƒ into contiguous blocks and move them before the embedding. The embedding then keeps its order inside each block and breaks adjacency only at the boundaries between blocks. Moving whole blocks in Table 6 is the discrete form of this experiment.

It is true that a continuous model gives good results at a level it was not trained on, and that this shows an inductive bias. But this bias comes from adjacency and not from the order. The embedding of an unseen level lies between those of the trained levels adjacent to it. Because the posterior means of these adjacent levels are close, interpolating between them gives a prediction close to the exact posterior mean. A reversed embedding keeps the same adjacent levels, so the same interpolation still works and the order does not matter. A shuffled embedding instead puts distant levels next to the unseen one, so the interpolation fails. We test this directly in the new Table 16, where we evaluate each mixture network at the levels between its trained ones, and the result agrees: the error at the unseen levels equals the error at the trained levels under the ordered and the reversed embedding, and is many times larger under a shuffled one.

If these address your concerns, we would be grateful if you could raise your score.

Reviewer_NDPkOpenReview โ†—
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Diffusion model is being trained across noise levels. However, their training objective is pointwise, where the loss is a sum over losses per noise level and the loss for each noise level does not interact with one another. Yet the model trained across noise levels performs better than models trained separately for each noise level that have the same total parameter counts. In contrast to methods that tie noise level with the training, which requires writing the order of the noise level into the objective itself, the author asks if the order of the noise levels is a design choice that is independent of how well the model can learn from the data.

The paper shows that the order of the noise levels has adjacent relation, meaning that noise levels adjacent to one another affect the errors overall. While the ordered and reversed noise levels show similar, almost identical, errors in transition, shuffled noise orders presents higher errors. The result also suggests that the direction of the noise levels does not impact the fitting error. Secondly, the paper suggests that the relation does not hold at distance. Overall, the study shows that the order of the noise levels in the data should be addressed before training, while the direction of the noise level remains a design choice.

Critical strengths

The paper investigates how the order of noise level enters a diffusion model from a novel perspective. Although the training objective is pointwise in the noise level, a shared model outperforms separate models with the same total parameter count. The study examines the question of the order of noise levels from direction and adjacency. Substantial experiments are done to validate that the adjacency in the noise level matters to the model performance while the direction does not. The training objective for a diffusion model stays unchanged since the sum over the local losses is invariant to the noise level assigned to each local loss.

Critical weaknesses

No substantial weaknesses.

Questions for authors

It will be interesting to see if the result is transferrable to other architecture backbone for the diffusion model.

2 comments

Thank you for acknowledging the contribution of our work. Our result is not limited to a specific backbone as our experiments already span several architectures. We additionally test our claim on order on a transformer (SiT on CIFAR-10) below (a second training in parentheses):

EmbeddingHeld-out denoising lossChange from ordered
ordered0.4598 (0.4619)-
reversed0.4634+0.8%
shuffled0.4759+3.5%

Thus, we can see that our results hold across simple MLP, U-Net and Transformer. We have added this table as the new row of Table 3 in the revised version.

Thank you for clarifying my question. I will maintain my score.

Comments on the paper

1 comment

We thank all reviewers for their careful reading and for acknowledging the soundness of the experiments. Since the remaining concerns are mainly about what the paper contributes, we state the contribution here and then list the changes in the revised version.

Contribution. The paper investigates how the order of the noise levels enters a diffusion model.

  • The question. The training objective is pointwise in the noise level, yet a shared model outperforms separate models with the same total parameter count. The order must therefore enter the model somewhere. Thus we ask where it enters.
  • The experiment. We demonstrated that the objective stays unchanged when the noise level assigned to each local loss changes in Theorem 1. We can therefore change only the noise-level embedding and keep the tasks fixed. Thus any change in the error must come from the embedding.
  • The result. We found that adjacency in the embedding is what matters to the performance instead of the direction.

This is also where our work differs from Go et al. (2023) and Park et al. (2024), who build methods on the observation that nearby timesteps are similar. Their observation is made under one fixed embedding. Under this embedding, noise levels with similar denoising tasks are also adjacent in the embedding, and the embedding always runs in the same direction, so the effects of task similarity, adjacency and direction cannot be separated while we separated them in our experiments.

The result is significant because previous work showed that much of the physical description of diffusion models is a design choice, but the order of the noise levels has remained fixed in every formulation. Now we show that its direction is also a design choice with adjacency as the key.

Changes in the revised version (marked in blue):

  • Table 16 (Appendix G, new): evaluation at noise levels that the network never saw.
  • Table 3 (new row): ordered, reversed and shuffled embeddings on a transformer (SiT).
  • Table 6 (new column): the reversed embedding on the mixture.
  • Section 6: the related work on Go et al. and Park et al. is moved to the main text.
  • Definition 1 defines the section f(ยท, k) and the posterior mean F_k before Theorem 1.
  • Section 3 defines the interleaved division before Table 1, and every table names its metric.
  • Tables 4, 5 and 6 give standard deviations over seeds.
  • Table 7: the results on retraining and replacing networks are moved to the main text.

We reply to each reviewer below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the mapโ€ฆ

Discussion (0)

Sign in to comment.