- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper investigates the reason why training one network acroll all noise levels in diffusion models performs better than training separate networks for each noise level. It is shown that the objectives are independent for each noise level k, and how the noise level embedding influences the shared network error. Numerical experiments illustrate that splitting the noise levels across separate networks matches or improves performance if the width is the same for each network, but deminishes performance if the total number of parameters is kept fixed, with better performance for contiguous ranges of noise levels. By experimentally analyzing the effects of reversing or shuffling the noise levels in the embedding, the authors conclude that adjacency and not direction is the key to increased performance.
Critical strengths
The experiments are rather extensive and test various setups.
Critical weaknesses
Three main problems with the paper: (1) previous studies already revealed that adjacency plays a key role in training a shared network for diffusion models, (2) the theoretical results are unsurprising and can be derived in a straightforward manner, and (3) the paper is not well written.
The closest prior work, Go et al. (2023) and Park et al. (2024), is not discussed in the main paper, and the results appear to overlap substantially with it. In particular, the papers already demonstrate that adjacency of noise levels plays a key role in training diffusion models. Go et al. observe that the task affinity between denoising tasks is high for adjacent noise levels and descreases smoothly as the difference increases. Park et al. propose a method that uses this task affinity by activating similar channels for tasks at adjacent timesteps and show that it consistently boosts performance. Against this background, the additional insights from the present paper seem minimal, diminishing its novelty and significance.
Theorem 1 is almost immediate and Lemma 1 follows directly from the assumptions.
Some key mathematical quantities in the paper are not properly defined (e.g. ๐น_๐, ๐, product of argmin). The paper is partly hard to read since Tables and Figures show results early on that are explained much later in the paper (e.g. Table 1 on page 4 includes results for โevery K-thโ which is not explained until Section 5 on page 8). The explanation with reference to Figure 2c on page 8 lines 424-426 is unclear. Moreover, the paper moves all related work to the appendix, which prevents readers from judging the contribution in context.
Questions for authors
What are the additional insights from the present paper compared to Go et al. (2023), and why are they significant?
Is there a reason/intuition why the graphs in Figure 6 have this shape?
When using several seeds in the experiments, it seams reasonable to also show the variance (not just the mean).
4 comments
Thank you for the response and the revised manuscript. The changes address part of my concerns. However, two concerns remain.
The first is the theoretical contribution, which I still consider minor. The rebuttal explains the motivation for the questions, but this does not change that Theorem 1 and Lemma 1 follow almost immediately from the setup, so the value of the paper lies in the experiments rather than the theory. You state that Lemma 1 predicts which embeddings fit worse, but the paper does not show that the bound tracks the empirical ranking of embeddings (e.g., contiguous vs. interleaved vs. shuffled). Such a comparison would make the theory more meaningful. Going further and quantifying in theory how the different setups affect the error (e.g., contiguous vs. interleaved splits, or the number of networks at a fixed parameter count) would be a valuable addition, though I realize this may be beyond the scope of the paper. I would also encourage stating more carefully that the evidence for direction being a design choice is limited to the tested setups.
The second concern is Figure 6. Your explanation accounts for the shape of the gain relative to a single network (narrower ranges fit better, and boundary transitions have neighbours on one side only). My question was about the loss pattern described in the manuscript, namely why the loss increases toward the clean end of the first three ranges and peaks at the noisy end of the last. The response does not explain this, and I would appreciate either an explanation or a clear statement that it is an empirical observation you do not yet understand.
Thank you for the constructive follow-up.
For the first concern, we agree that the main value of the paper lies in the experiments, but the experiments depend on the theory. Theorem 1 is what allows us to change the embedding while the objective stays the same, so that a change in the error can only come from the network. Lemma 1 gives a bound that is computed from the data before any training, and it should predict which embeddings fit worse. To test this, we ran the comparison you suggest. We rank the embeddings (11 on the mixture, the 6 of Table 3 on CIFAR-10) by the bound and compare this ranking with the ranking by the measured error:
| Spearman correlation between the two rankings | |
|---|---|
| mixture, width 256 | 0.86 |
| mixture, width 512 | 0.86 |
| mixture, width 1024 | 0.27 |
| CIFAR-10 | 0.99 |
The two rankings agree well on the mixture at widths 256 and 512 and on CIFAR-10. At width 1024 the correlation is low because all embeddings except the full shuffle have almost the same error, with differences comparable to the variation across seeds. For the divisions you mention, the bound is lowest for contiguous ranges, does not increase for interleaved ones, and is highest for a shuffle, and the measured errors follow the same order. Thus we can conclude that the bound does track the empirical ranking. Quantifying these effects in theory at a fixed parameter count is beyond the scope of this paper, as you note.
We agree that the evidence for direction being a design choice is limited to the settings we test, and we now state this as a limitation in the conclusion.
For the second concern, the pattern comes from two effects. First, a transition next to a boundary has neighbours only on one side, so its loss relative to a single network is higher (Table 2). Second, in the first and the last range, the gain from separate networks is larger towards the end of the grid, where EDM's training distribution has few samples and a single network is least trained. In the first range both effects point to the clean end, and in the last range both point to the noisy end, which is why the loss is highest there. The two middle ranges have a boundary at both ends, but their loss rises mainly towards the clean end. This asymmetry is the only part that we cannot yet explain.
We hope this addresses your concerns.
Thanks for the response; following up on the discussion, I would raise my score to 2.
Thank you for acknowledging that our experiments are extensive and test various setups. Table numbers refer to the revised version.
W1 and Q1: Go et al. (2023) and Park et al. (2024) build methods on the observation that nearby timesteps are similar. We ask a different question: the training objective does not contain an order of the noise levels (Theorem 1), so where can an order enter a model, and which part of it does the network use? Their observation cannot answer this question, because it is made under one fixed embedding. Under this embedding, noise levels with similar denoising tasks are also adjacent in the embedding, and the embedding always runs in the same direction, so the effects of task similarity, adjacency and direction cannot be separated. For this reason, we only change the embedding and keep the tasks fixed. Since the tasks are the same, any change in the error must come from the embedding, and we find that it depends on which levels are adjacent and not on their direction. This result is thus novel.
It is significant because many previous work showed that much of the physical description of diffusion models is a design choice, but the order of the noise levels has remained fixed in every formulation. We show that only its adjacency is needed, so its direction is also a design choice. We now discuss both works in Section 6.
W2: Our proofs are short, but this does not mean that the question itself is simple. Diffusion models are formulated along a time axis, yet Theorem 1 shows that their objective treats the noise levels as an unordered family. So if the order matters, it can only enter through the network, which motivates our question. It also means that relabelling the embeddings keeps the objective fixed (Corollary 1), so a change in the error after relabelling can only come from the network.
Lemma 1 uses one triangle inequality. Its contribution lies in the choice of quantities: it expresses the fitting error in terms of the distances between posterior means. Because these distances are determined by the data before any training, the bound predicts which embeddings fit worse.
W3: We now define the section f(ยท, k) and the posterior mean F_k in Definition 1 and write out the product of argmin in Theorem 1. We also define the interleaved (every K-th) division in Section 3 before Table 1, rewrite the passage on Figure 2c, and move the closest related work to the main text (Section 6).
Q2: The shape comes from two effects. First, a network that covers a narrower range at the same width fits its levels better, so the curves lie below zero over most of the grid. Second, a transition next to a boundary gains less, because its network contains its neighbours on one side only (Table 2). The grey line separates the two effects: networks that each cover every 4th transition get no narrowing, and they stay close to one network everywhere. This is why the gain is smallest next to the boundaries and largest inside the ranges.
Q3: We add the standard deviation beside every mean over seeds in Tables 4, 5 and 6.
If these address your concerns, we would be grateful if you could raise your score.