acceptodds
Desk rejected at ICLR 2027

NoiseRater: Meta-Learned Noise Valuation for Diffusion Model Training

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Training a diffusion model involves two sources of randomness for each data sample: the timestep and the Gaussian noise realization. The timestep has been studied extensively through scheduling and weighting, whereas the impact of the noise realization at a given timestep is still underexplored. In this work, we examine whether different noise instances are equally informative. We introduce NoiseRater, a network that scores an individual noise instance conditioned on the data sample and timestep. The rater is learned through bilevel optimization, where its scores reweight the diffusion loss in the inner loop and it is updated to reduce validation loss after the inner-loop updates. Using the trained rater to select training noise, we observe three properties of training noise. First, noise realizations at the same timestep are not equally useful, with the rater's top-scored noise improving performance over i.i.d. sampling and its bottom-scored noise degrading it. Second, this utility is contextual, depending jointly on the image, the class, and the timestep. Third, noise selection is complementary to timestep-level design, retaining most of its gain when combined with existing scheduling and weighting schemes. These findings establish instance-level noise valuation as a new axis for understanding and improving diffusion training. Code is available at https://anonymous.4open.science/r/NoiseRater-8ECB.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
meta learning, diffusion model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AzfHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether giving different weights to different noises can help diffusion model training. The authors first train a noise rater at the beginning of training to evaluate each noise. Then after fixing the evaluator, each step generates KK noises and use the one with highest score. Empirical experiments show modest improvements of this strategy.

Critical strengths
  1. Studying noise selection during training is a worthwhile research direction.

  2. The controlled experiments are useful and informative.

  3. The paper is well-written and easy to read.

Critical weaknesses

I find the empirical results not compelling enough. The improvements (e.g. Table 1 on ImageNET) all look very marginal. The method does not look very scalable (e.g., the improvement is negligible on DiT-L/2).

The compute-matched comparison result does not include the meta-learning cost.

Therefore I appreciate the results, and I think it's worth studying. However the main information I get after reading the paper seems to be "we should probably not trying to spend much effort in selecting noises during training".

I'd love to hear the authors' thoughts.

Questions for authors

Can you provide a compute-matched comparison that includes the meta-training cost?

How concentrated are the learned weights for the noises?

4 comments
AuthorsOpenReview ↗

Response to Reviewer AzfH (1/2)

Thank you for asking whether the FID gains are large enough to justify the added work. We share your concern about the weak result on DiT-L/2 and agree that the default rater’s cost matters for a single run. Your question about the weights also gave us a chance to show how much of the weight falls on the top candidate.

[W1.1] Size of the improvements

We agree that the absolute FID improvements are modest. On ImageNet DiT-S/2, we repeated the +80k-step vanilla and full-rater continuations with three seeds. For the p=4 rater, we separately varied the continuation seed with the rater fixed and the meta-training seed with the continuation seed fixed. We also repeated the comparisons with three timestep-level methods.

The mean improvements are 1.42–1.71 FID over vanilla and 1.21–1.27 FID when adding the rater to the timestep-level methods. The reported run-to-run standard deviations are 0.02–0.13 FID. These experiments support repeatability in the tested DiT-S/2 settings. Table 1 also shows a 4.46 FID gap between the top-1 and bottom-1 selection policies (59.97 versus 64.43), with a 1.72 FID improvement for top-1 over vanilla.

ConfigurationFID-50K ↓ (mean ± std)Average gain over baseline
Vanilla61.67 ± 0.02—
NoiseRater (full), K=4, top-159.96 ± 0.031.71
NoiseRater (small, patch 4), K=4, top-1, varying DiT seed60.25 ± 0.131.42
NoiseRater (small, patch 4), K=4, top-1, varying rater seed60.24 ± 0.031.43
ConfigurationFID-50K ↓ (mean ± std)Average gain over baseline
Min-SNR59.08 ± 0.08—
Min-SNR + NoiseRater (small)57.86 ± 0.111.22
P257.48 ± 0.12—
P2 + NoiseRater (small)56.21 ± 0.091.27
Logit-normal57.23 ± 0.06—
Logit-normal + NoiseRater (small)56.02 ± 0.041.21

[W1.2] Scaling to DiT-L/2

The gain on DiT-L/2 is negligible, as you noted. The detail we should have made clearer is what we transferred. Appendix E tests whether a rater meta-trained on DiT-S/2 transfers directly to larger backbones at K=2. This transfer improves DiT-B/2 from 48.65 to 47.80 FID at +80k steps, and the gain disappears on DiT-L/2. We did not meta-train a rater on DiT-L/2. We will describe the result as a limitation of the tested cross-backbone transfer and leave efficacy and cost at that scale open.

[W2 & Q1] Compute-matched comparison including the meta-training cost

You are right that the original comparison left out a cost that matters for a single run. Table 6 charges the scoring overhead during continuation. With the 11.6 GPU-hours of rater meta-training included, the default S=5, p=4 rater gives 60.21 FID versus 60.19 for vanilla at about 31 GPU-hours from the shared 400k-step checkpoint. At the smaller budgets we examined, it trails vanilla by 0.1–0.3 FID.

Appendix F motivated testing one inner step with the same p=4, h=384, d=12 architecture. This reduces meta-training from 11.6 to 3.1 GPU-hours, with +80k FID of 60.18 for S=1 and 60.21 for S=5. The table charges meta-training and continuation, including scoring (4.85 GPU-hours per 20k steps versus 4.40 for vanilla). At each listed budget, S=1 has lower FID in this run. It reaches 60.18 FID at 22.5 GPU-hours, while vanilla reaches a similar 60.19 at 30.8 GPU-hours, about 27% less compute for NoiseRater in this continuation comparison. Both methods start from the same checkpoint, whose pretraining cost is shared and excluded. The S=1 numbers come from one run.

Total compute (GPU-h)NoiseRater (S=1), meta-training includedVanilla at the same total computeGain
7.961.8963.051.16
12.861.3462.330.99
17.660.7361.700.97
22.560.1861.110.93
AuthorsOpenReview ↗

Response to Reviewer AzfH (2/2)

[W3] The takeaway of the paper

We agree that the current method offers modest gains and that the cost matters. At equal continuation steps on ImageNet DiT-S/2, the same rater yields 59.97 FID when it selects the highest-scoring candidate and 64.43 when it selects the lowest-scoring candidate, compared with 61.69 for vanilla. The three timestep-level methods we tested also benefit from adding the rater. The input ablations show that the ranking's downstream benefit is particularly sensitive to timestep conditioning.

The practical case depends on the setting. Charging meta-training removes the single-run compute advantage for the default S=5 configuration at the budgets we examined. The cheaper S=1 rater reaches similar FID with roughly 27% less continuation compute from the shared checkpoint, including meta-training, in one run. The results support studying noise selection as an additional training choice, with its value depending on rater cost, coverage, and transfer to the target backbone.

[Q2] Concentration of the learned weights

The weight on the top candidate depends strongly on pool size in our measurements. We evaluated the weight concentration on 2,048 held-out ImageNet images at 10 timesteps using three raters: the full-size rater, its S=1 variant, and the p=4 rater. The mean top-candidate weight is 0.94 for K=2, 0.74 for K=4, and 0.59 for K=8. The corresponding effective candidate counts, measured by exp(H) with H the entropy of the group weights, are 1.2, 2.0, and 3.3.

These statistics describe the soft weights used by the meta-training objective. In the default continuation, the frozen rater supplies a ranking and only the top candidate is used.

KMean weight on the top candidateEffective number of candidates, exp(H)Uniform weight
20.941.20.50
40.742.00.25
80.593.30.125

We appreciate the distinction you drew between an interesting research question and a method worth adopting. We hope the repeatability results and the smaller rater’s cost comparison make the case for the finding at the scale we tested. If that affects your overall assessment, we would be grateful if you would reconsider your score.

I appreciate the authors' detailed response. The additional experiments are informative and helpful. Nevertheless, they seem to reinforce my concern on the practical usefulness and scalability of noise rating?

That said, I appreciated the authors' honesty and effort. At the same time, I think this is a very reasonable direction even if experiments show marginal and modest gains. I therefore increase my score to 3. I'd suggest the authors to tighten the contribution in the revised version.

AuthorsOpenReview ↗

Appreciate your recognition

Thank you for taking the time to consider the additional results and for revisiting your assessment. We appreciate your suggestion to sharpen the contribution. In the revision, we’ll center the abstract and conclusion on our main empirical finding: the choice of noise realization matters for diffusion training, and learned selection yields modest but consistent FID gains in the DiT-S/2 settings we tested. We’ll make the scope of that finding clear throughout the paper.

Reviewer_JF2qOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors introduce NoiseRater, a network that scores individual noise instances conditioned on the data sample and denoising timesteps. The model is trained through bilevel optimization: its scores reweight diffusion losses in the inner loop, while the outer loop optimizes the rater using held-out validation loss after the model updates. Once trained, the rater is frozen and selects the highest-scoring noise from multiple Gaussian candidates for continued diffusion training. Experiments with DiT models on FFHQ and ImageNet show the improvement on FID over random sampling.

Critical strengths
  1. Table 1 shows that different noise realizations vary in their usefulness for diffusion model training, highlighting the value of selecting informative training noise.

  2. Noise selection complements timestep-level design, providing additional improvements when combined with existing timestep weighting and sampling strategies.

Critical weaknesses
  1. Selecting noise based on the data alters the noise distribution during the training phase, while the sampler remains unchanged. This can lead to a biased denoiser that deviates from the target objective—which is defined based on standard Gaussian noise.
  2. As shown in Table 6, the performance gains under comparable computational resource constraints (17.6 GPU hours) are relatively limited (an FID improvement of 0.70–1.26). And the paper does not report the variance across multiple independent runs or provide confidence intervals.
Questions for authors
  1. How does NoiseRater compare with a baseline trained using the same computational budget in denoising error under standard Gaussian corruption on unseen images? How do these results change when models trained on ImageNet are evaluated on a related dataset that was not used during training?

  2. What are the mean and standard deviation of the FID scores in Table 6 across independent training runs? Do confidence intervals for the differences between NoiseRater and the baselines indicate that the improvements reliably exceed random fluctuations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer JF2q

Thank you for suggesting that we test the denoiser on ordinary Gaussian noise, including on a related dataset. That gave us a concrete way to check what happens when we evaluate the models under the original corruption process. We also took your concern about the size of the FID gains seriously and repeated the comparisons at the budget you highlighted.

[W1 & Q1] Training-noise selection alters the noise distribution

We agree that selecting noise conditional on the data changes the training objective. For any measurable set of noise realizations, the selected draw can belong to that set only if at least one of the K Gaussian candidates belongs to it. The selected density is therefore at most K times the standard Gaussian density. This bound alone does not establish that the objectives are close. The sampler continues to use the standard procedure.

We evaluated the shared 400k-step checkpoint and both +80k-step DiT-S/2 continuations on ImageNet validation (50,000 images) and ImageNet-V2 (10,000 images). For each comparison, we used identical image, timestep, and standard Gaussian noise triples, with no noise selection at evaluation. We were able to match the number of continuation steps for this comparison. The requested GPU-hour-matched MSE comparison remains open.

The NoiseRater continuation has MSE 0.00013 higher than the vanilla continuation on each dataset, about 0.09% relative, and both continuations improve over the shared starting checkpoint. This shows a small difference in average Gaussian-corruption MSE at the tested checkpoints, including on ImageNet-V2. Under the unchanged sampler, we also observe lower FID across three seeds (59.96 ± 0.03 versus 61.67 ± 0.02). Hard selection can still reduce recall and coverage, which we will report as a limitation.

ModelImageNet val (50k)ImageNet-V2 (10k)
Starting checkpoint (400k)0.143320.14620
Vanilla, +80k0.143030.14592
NoiseRater (K=4), +80k0.143160.14605

[W2 & Q2] Variance across independent runs and confidence intervals for Table 6

We repeated the four comparisons at the 17.6 GPU-hour point of Table 6, using the same p=4, h=384, d=12 rater. This budget includes continuation and scoring but excludes rater meta-training, as in the original table. Across the tested configurations, the mean FID improvements are approximately 0.8–1.2, with within-method standard deviations of 0.02–0.12. The repeated runs support the direction of the observed gains in these comparisons. We report the observed mean and standard deviation below. These summary statistics do not establish a confidence interval for the FID difference.

Method (17.6 GPU-h)Baseline FID ↓ (mean ± std)+ NoiseRater FID ↓ (mean ± std)Gain
Vanilla61.67 ± 0.0260.48 ± 0.101.19
Min-SNR59.08 ± 0.0858.22 ± 0.110.86
P257.48 ± 0.1256.65 ± 0.080.83
Logit-normal57.23 ± 0.0656.38 ± 0.080.85

Thank you again for suggesting these checks. Our denoising comparison covers both datasets at equal steps, and the repeated FID runs address a separate question at equal GPU-hours. If these results give you more confidence in the empirical findings, we would appreciate your reconsidering the score.

Thank you for the additional experiments. It appears that NoiseRater can generalize to unseen noise distributions. Although the underlying mechanism remains unclear (i.e., regarding the mismatch between training and inference), I am happy to raise my score.

We really appreciate your recognition of our work. Thank you for your follow-up and for raising your score. We will further study how training-time noise selection shapes the learned denoiser, and we hope this work can inspire more research on training-time noise in diffusion models.

Reviewer_YTdtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces NoiseRater, a meta-learned network that scores candidate Gaussian noise realizations for diffusion training. The rater is trained through bilevel optimization and then used to select one noise sample from a candidate pool. Experiments on FFHQ and ImageNet report improved FID over random noise selection and compatibility with timestep weighting and scheduling methods.

Critical strengths
  1. The paper identifies a novel and potentially useful axis of diffusion training: selecting individual noise realizations rather than only choosing or weighting timesteps.

  2. Top- versus bottom-rated noise, candidate-pool, compute-matched, and timestep-method experiments consistently indicate that the learned ranking contains a useful training signal.

Critical weaknesses
  1. The central empirical evidence lacks independent runs or uncertainty estimates. Several improvements are small, and checkpoints from one run are not independent evidence. Stronger selection baselines, such as loss- or gradient-based proxies, are also missing.

  2. The contextual-dependence and generality claims are overstated. The class-only ablation is missing, and the nominally unconditional ImageNet result appears to let the rater use class labels. The method also changes the Gaussian training-noise distribution without evaluating diversity or coverage, while matched-compute results exclude the substantial one-time rater-training cost.

Questions for authors
  1. Does the rater use ImageNet class labels in the unconditional experiments? If so, can you report a comparison using a label-free rater?

  2. Can you report results over multiple seeds with uncertainty, including precision/recall or coverage metrics, and include rater meta-training in total compute?

2 comments
AuthorsOpenReview ↗

Response to Reviewer YTdt (1/2)

Thank you for looking closely at how we used class labels and what we counted as compute. You were right to flag the ImageNet label: our wording made it easy to read the setting as fully unconditional. Your questions about seeds and coverage also prompted us to look at how repeatable the gains are and what hard selection costs in diversity.

[W1.1 & Q2.1] Empirical evidence with uncertainty estimates

We repeated the ImageNet DiT-S/2 continuation with three seeds per configuration for vanilla training and for the full-size rater. For the p=4 rater, we varied the DiT continuation seed with the rater fixed, and separately varied the rater meta-training seed with the continuation seed fixed. We also repeated the timestep-method comparisons. The tables report FID-50K at +80k continuation steps.

The mean gains are 1.42–1.71 FID for the plain continuation and 1.21–1.27 FID when adding the rater to the three timestep-level methods. The reported run-to-run standard deviations are 0.02–0.13 FID. These results support repeatability in the tested ImageNet DiT-S/2 settings. The two p=4 seed sweeps measure sensitivity to each stage separately.

ConfigurationFID-50K ↓ (mean ± std)Average gain over baseline
Vanilla61.67 ± 0.02—
NoiseRater (full), K=4, top-159.96 ± 0.031.71
NoiseRater (small, patch 4), K=4, top-1, varying DiT seed60.25 ± 0.131.42
NoiseRater (small, patch 4), K=4, top-1, varying rater seed60.24 ± 0.031.43
ConfigurationFID-50K ↓ (mean ± std)Average gain over baseline
Min-SNR59.08 ± 0.08—
Min-SNR + NoiseRater (small)57.86 ± 0.111.22
P257.48 ± 0.12—
P2 + NoiseRater (small)56.21 ± 0.091.27
Logit-normal57.23 ± 0.06—
Logit-normal + NoiseRater (small)56.02 ± 0.041.21

[W1.2] Loss- or gradient-based proxies for noise selection

Table 1 includes two heuristics under the same K=4, keep-one protocol. On ImageNet without CFG, selecting the largest- and smallest-norm candidates gives 60.82 and 63.31 FID at +80k steps, respectively. Vanilla gives 61.69 and NoiseRater gives 59.97.

We agree that a stronger baseline would help put the gain in context. Our meta-objective scores the contribution of a candidate through held-out loss after an inner-loop update. A direct selection rule based on the candidate's current loss or gradient norm would be a useful comparison. The current evaluation has neither baseline, so the evidence establishes an advantage over random selection and the two norm heuristics in this setting.

[W2.1 & Q1] Class labels in the "unconditional" setting and a label-free rater

Yes, the rater receives the class label. The original “ImageNet, Uncond.” row uses a class-conditional DiT and a rater with access to the class label. Sampling uses the class condition with CFG scale 1.0. We will relabel the setting “ImageNet, class-conditional without CFG.”

We meta-trained a rater using the image and timestep as context, with the explicit class-label input removed, and evaluated it under the same K=4, top-1 protocol. The denoiser and sampling protocol retain their class conditioning. At +80k steps, the label-free rater improves FID by 1.59, versus 1.72 for the full rater. FFHQ provides a separate setting with no class labels, where Table 1 also shows a gain over vanilla.

Removing the timestep produces the largest degradation in Table 2, from 59.97 to 61.40 FID. Removing the image while retaining timestep and class gives 60.29. Removing only the class label gives 60.10 in the new comparison. Table 2 does not include a rater with class as its only context input. We will narrow the manuscript's claim to the observed effects of these input ablations, with the strongest evidence for timestep conditioning.

Method+20k+40k+60k+80kGain over Vanilla at +80k
Vanilla63.6062.9362.2661.69—
NoiseRater, full (image + class + timestep)61.8861.0360.3759.971.72
NoiseRater, label-free (image + timestep)61.8861.2560.7260.101.59
AuthorsOpenReview ↗

Response to Reviewer YTdt (2/2)

[W2.2 & Q2.2] Diversity and coverage under noise selection

Your question about diversity surfaced a trade-off that FID alone hid. At +80k steps, hard top-1 selection improves precision and density at the cost of recall and coverage in this comparison. With K=4, for example, precision rises from 0.725 to 0.741 while recall falls from 0.232 to 0.207 and coverage from 0.459 to 0.438. K=8 shows a larger decline in recall and coverage and worse FID than K=4. Bottom-1 selection is below vanilla on all reported metrics, including both precision and recall.

Soft weighting over all four candidates gives 59.42 FID, with recall of 0.221 and coverage of 0.453, closer to vanilla. This variant uses forward and backward computation for all four candidates and is more expensive per step. We will report the diversity and coverage trade-off alongside the FID gains and distinguish the soft-weighting result from the default hard selection.

MethodFID ↓Precision ↑Recall ↑Density ↑Coverage ↑IS ↑
Vanilla (i.i.d. noise)61.670.7250.2321.0620.45919.63
NoiseRater, K=2, top-160.380.7340.2241.0940.45519.60
NoiseRater, K=4, top-159.970.7410.2071.1520.43819.08
NoiseRater, K=8, top-160.970.7530.1911.2690.41117.92
NoiseRater, K=4, bottom-164.430.7200.2261.0400.44618.86
NoiseRater, K=4, soft weighting over all K59.420.7390.2211.1190.45319.71

[W2.3 & Q2.3] Compute comparison including the rater meta-training cost

You are right that the original comparison left out a cost that matters for a single run. Table 6 charges the scoring overhead during continuation. With the 11.6 GPU-hours of rater meta-training included, the default S=5, p=4 rater gives 60.21 FID versus 60.19 for vanilla at about 31 GPU-hours from the shared 400k-step checkpoint. At the smaller budgets we examined, it trails vanilla by 0.1–0.3 FID.

Appendix F motivated testing one inner step with the same p=4, h=384, d=12 architecture. This reduces meta-training from 11.6 to 3.1 GPU-hours, with +80k FID of 60.18 for S=1 and 60.21 for S=5. The table charges meta-training and continuation, including scoring (4.85 GPU-hours per 20k steps versus 4.40 for vanilla). At each listed budget, S=1 has lower FID in this run. It reaches 60.18 FID at 22.5 GPU-hours, while vanilla reaches a similar 60.19 at 30.8 GPU-hours, about 27% less compute for NoiseRater in this continuation comparison. Both methods start from the same checkpoint, whose pretraining cost is shared and excluded. The S=1 numbers come from one run.

Total compute (GPU-h)NoiseRater (S=1), meta-training includedVanilla at the same total computeGain
7.961.8963.051.16
12.861.3462.330.99
17.660.7361.700.97
22.560.1861.110.93

We appreciate how specific your review was. We hope the repeated runs, label-free comparison, and coverage results address the main concerns behind your assessment. If these results change your view of the evidence, we would be grateful if you would reconsider your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Meta-response

Dear Area Chair and Reviewers,

Thank you for giving NoiseRater a careful reading. Your questions helped us sharpen both the evidence and the claims. We agree that the absolute gains are modest and that the practical value of noise selection depends on its compute cost, its effect on coverage, and how well the rater transfers to the target backbone.

  • Repeatability and conditioning. In three ImageNet DiT-S/2 continuations, FID-50K was 59.96 ± 0.03 with the full rater and 61.67 ± 0.02 with vanilla training (mean ± SD at +80k steps). We also observed gains when adding the rater to each of the three timestep methods tested. Removing the explicit class label from the rater retained 1.59 of the original 1.72 FID improvement. The denoiser remains class-conditional in this comparison, and we will relabel the original setting as class-conditional without CFG.
  • Denoising and coverage. Under standard Gaussian corruption, the rater-trained model’s average MSE was about 0.09% higher than vanilla on both ImageNet validation and ImageNet-V2 at equal continuation steps. A GPU-hour-matched MSE comparison remains open. Hard top-1 selection improved precision while reducing recall and coverage. At K=4, recall fell from 0.232 to 0.207. We will report this trade-off alongside the FID gains.
  • Compute and scale. After charging meta-training, the default S=5 rater was roughly on par with vanilla at about 31 GPU-hours of continuation compute. In one run, the cheaper S=1 rater reached 60.18 FID at 22.5 GPU-hours, while vanilla reached a similar 60.19 at 30.8 GPU-hours. Both totals start from the shared pretrained checkpoint and include all costs after that point. The rater trained on DiT-S/2 provided negligible benefit when transferred to DiT-L/2, so the evidence for larger backbones remains limited.

We will narrow the manuscript to the finding these experiments support: which noise realization is selected can affect downstream training, and learned selection gives a modest, repeatable gain in the tested DiT-S/2 settings.

We appreciate the time you spent on these questions. We hope the added evidence and clearer scope help in weighing the contribution. If they change your assessment, we would be grateful if you would reconsider your recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.