- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether giving different weights to different noises can help diffusion model training. The authors first train a noise rater at the beginning of training to evaluate each noise. Then after fixing the evaluator, each step generates noises and use the one with highest score. Empirical experiments show modest improvements of this strategy.
Critical strengths
-
Studying noise selection during training is a worthwhile research direction.
-
The controlled experiments are useful and informative.
-
The paper is well-written and easy to read.
Critical weaknesses
I find the empirical results not compelling enough. The improvements (e.g. Table 1 on ImageNET) all look very marginal. The method does not look very scalable (e.g., the improvement is negligible on DiT-L/2).
The compute-matched comparison result does not include the meta-learning cost.
Therefore I appreciate the results, and I think it's worth studying. However the main information I get after reading the paper seems to be "we should probably not trying to spend much effort in selecting noises during training".
I'd love to hear the authors' thoughts.
Questions for authors
Can you provide a compute-matched comparison that includes the meta-training cost?
How concentrated are the learned weights for the noises?
4 comments
Response to Reviewer AzfH (2/2)
[W3] The takeaway of the paper
We agree that the current method offers modest gains and that the cost matters. At equal continuation steps on ImageNet DiT-S/2, the same rater yields 59.97 FID when it selects the highest-scoring candidate and 64.43 when it selects the lowest-scoring candidate, compared with 61.69 for vanilla. The three timestep-level methods we tested also benefit from adding the rater. The input ablations show that the ranking's downstream benefit is particularly sensitive to timestep conditioning.
The practical case depends on the setting. Charging meta-training removes the single-run compute advantage for the default S=5 configuration at the budgets we examined. The cheaper S=1 rater reaches similar FID with roughly 27% less continuation compute from the shared checkpoint, including meta-training, in one run. The results support studying noise selection as an additional training choice, with its value depending on rater cost, coverage, and transfer to the target backbone.
[Q2] Concentration of the learned weights
The weight on the top candidate depends strongly on pool size in our measurements. We evaluated the weight concentration on 2,048 held-out ImageNet images at 10 timesteps using three raters: the full-size rater, its S=1 variant, and the p=4 rater. The mean top-candidate weight is 0.94 for K=2, 0.74 for K=4, and 0.59 for K=8. The corresponding effective candidate counts, measured by exp(H) with H the entropy of the group weights, are 1.2, 2.0, and 3.3.
These statistics describe the soft weights used by the meta-training objective. In the default continuation, the frozen rater supplies a ranking and only the top candidate is used.
| K | Mean weight on the top candidate | Effective number of candidates, exp(H) | Uniform weight |
|---|---|---|---|
| 2 | 0.94 | 1.2 | 0.50 |
| 4 | 0.74 | 2.0 | 0.25 |
| 8 | 0.59 | 3.3 | 0.125 |
We appreciate the distinction you drew between an interesting research question and a method worth adopting. We hope the repeatability results and the smaller rater’s cost comparison make the case for the finding at the scale we tested. If that affects your overall assessment, we would be grateful if you would reconsider your score.
I appreciate the authors' detailed response. The additional experiments are informative and helpful. Nevertheless, they seem to reinforce my concern on the practical usefulness and scalability of noise rating?
That said, I appreciated the authors' honesty and effort. At the same time, I think this is a very reasonable direction even if experiments show marginal and modest gains. I therefore increase my score to 3. I'd suggest the authors to tighten the contribution in the revised version.
Appreciate your recognition
Thank you for taking the time to consider the additional results and for revisiting your assessment. We appreciate your suggestion to sharpen the contribution. In the revision, we’ll center the abstract and conclusion on our main empirical finding: the choice of noise realization matters for diffusion training, and learned selection yields modest but consistent FID gains in the DiT-S/2 settings we tested. We’ll make the scope of that finding clear throughout the paper.
Response to Reviewer AzfH (1/2)
Thank you for asking whether the FID gains are large enough to justify the added work. We share your concern about the weak result on DiT-L/2 and agree that the default rater’s cost matters for a single run. Your question about the weights also gave us a chance to show how much of the weight falls on the top candidate.
[W1.1] Size of the improvements
We agree that the absolute FID improvements are modest. On ImageNet DiT-S/2, we repeated the +80k-step vanilla and full-rater continuations with three seeds. For the p=4 rater, we separately varied the continuation seed with the rater fixed and the meta-training seed with the continuation seed fixed. We also repeated the comparisons with three timestep-level methods.
The mean improvements are 1.42–1.71 FID over vanilla and 1.21–1.27 FID when adding the rater to the timestep-level methods. The reported run-to-run standard deviations are 0.02–0.13 FID. These experiments support repeatability in the tested DiT-S/2 settings. Table 1 also shows a 4.46 FID gap between the top-1 and bottom-1 selection policies (59.97 versus 64.43), with a 1.72 FID improvement for top-1 over vanilla.
[W1.2] Scaling to DiT-L/2
The gain on DiT-L/2 is negligible, as you noted. The detail we should have made clearer is what we transferred. Appendix E tests whether a rater meta-trained on DiT-S/2 transfers directly to larger backbones at K=2. This transfer improves DiT-B/2 from 48.65 to 47.80 FID at +80k steps, and the gain disappears on DiT-L/2. We did not meta-train a rater on DiT-L/2. We will describe the result as a limitation of the tested cross-backbone transfer and leave efficacy and cost at that scale open.
[W2 & Q1] Compute-matched comparison including the meta-training cost
You are right that the original comparison left out a cost that matters for a single run. Table 6 charges the scoring overhead during continuation. With the 11.6 GPU-hours of rater meta-training included, the default S=5, p=4 rater gives 60.21 FID versus 60.19 for vanilla at about 31 GPU-hours from the shared 400k-step checkpoint. At the smaller budgets we examined, it trails vanilla by 0.1–0.3 FID.
Appendix F motivated testing one inner step with the same p=4, h=384, d=12 architecture. This reduces meta-training from 11.6 to 3.1 GPU-hours, with +80k FID of 60.18 for S=1 and 60.21 for S=5. The table charges meta-training and continuation, including scoring (4.85 GPU-hours per 20k steps versus 4.40 for vanilla). At each listed budget, S=1 has lower FID in this run. It reaches 60.18 FID at 22.5 GPU-hours, while vanilla reaches a similar 60.19 at 30.8 GPU-hours, about 27% less compute for NoiseRater in this continuation comparison. Both methods start from the same checkpoint, whose pretraining cost is shared and excluded. The S=1 numbers come from one run.