- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates why likelihood-weighted diffusion classifiers fail when faced with domain shift. The authors decomposed class scores into components corresponding to different noise levels, discovering that low-noise terms can bias the classifier toward the wrong class when images undergo rotation or are subjected to noise interference. Consequently, the authors proposed restricting classification to specific noise intervals and averaging the denoising errors within those intervals—an approach that requires no model retraining. This method improved accuracy on the Rotated MNIST dataset from 57.9% to 92.7% and on the CIFAR-10-C noisy dataset from 36.3% to 77.8%, although performance declined for certain other types of shifts.
Critical strengths
- This paper identifies a specific failure mechanism: in the presence of domain shift, low-noise evidence can be misleading, and likelihood-based weighting amplifies this effect.
- Comparative experiments utilized identical models and denoising errors, enabling an isolated assessment of the impact of scoring rules; experiments involving "bucket removal" and "finer integration" further corroborate this analysis.
- The proposed solution is simple to implement and requires no model retraining. Additionally, the paper outlines scenarios where the method fails, thereby clarifying its scope of applicability and the limits of its benefits.
Critical weaknesses
-
The experiments are limited to small, class-conditioned U-Net models trained on the MNIST and CIFAR datasets. It remains unclear whether the "low-noise failure" phenomenon mentioned in the paper exists in larger-scale pre-trained models or generalizes to text-to-image models or SIT weights. Consequently, the primary issue with this paper stems from the inherent limitations of the diffusion classifier itself.
-
The analysis above suggests the existence of a baseline method superior to simply discarding unreliable "buckets." If certain buckets are systematically biased toward incorrect classes, their outputs can serve as negative references in contrastive logit extrapolation, specifically: . When "good buckets" and "bad buckets" correspond to the margins of the correct class in the positive and negative directions, respectively, this approach can yield further performance improvements compared to using only "good buckets"; the authors might consider verifying the feasibility of this method.
Questions for authors
No
7 comments
Response to Q1 of Reviewer WzfU (2/3)
(Continued, part 2 of 3.)
Larger pretrained models (run during the discussion period). We scored two ImageNet models that we did not train, on the same 500 ImageNet validation images (the 10 Imagenette classes, 50 per class; 10-way classification with the 1000-class model), clean plus 14 corrupted settings (Gaussian, shot and impulse noise, fog, contrast, defocus blur, JPEG; severities 3 and 5; ImageNet-C / Tiny-ImageNet-C constants). The pipeline is unchanged: the same 16 log-SNR buckets, shared (t, ε) pairs and ITD quadrature weights. (a) ImageNet-64 EDM (Karras et al. 2022; ADM backbone, 296M parameters, pixel space). (b) SiT-XL/2 at 256×256 (Ma et al. 2024; 675M parameters, latent space, velocity prediction; the corruption is applied to the image, which is then encoded by the VAE; the buckets are mapped to the interpolant time through log-SNR).
| Score (N = 500 per setting) | clean | noise family (6 settings) | non-noise (8 settings) | all 14 |
|---|---|---|---|---|
| ImageNet-64 EDM, exact likelihood (ITD) | 96.4 | 18.4 | 69.7 | 47.7 |
| ImageNet-64 EDM, uniform-t | 97.0 | 28.4 | 71.2 | 52.8 |
| ImageNet-64 EDM, band 2–15 (the paper's locked rule) | 96.4 | 32.5 | 66.4 | 51.9 |
| ImageNet-64 EDM, band 3–15 (the paper's hold-out band) | 95.4 | 40.2 | 64.7 | 54.2 |
| ImageNet-64 EDM, band selected on the other corruption types (7–15 to 9–15) | 93–94 | 83.8 | 55.0 | 67.3 |
| SiT-XL/2, exact likelihood (ITD) | 95.4 | 59.5 | 64.9 | 62.6 |
| SiT-XL/2, uniform-t | 97.4 | 54.6 | 69.9 | 63.3 |
| SiT-XL/2, band 2–15 | 97.4 | 52.7 | 70.5 | 62.9 |
| SiT-XL/2, band 3–15 | 97.0 | 50.6 | 69.8 | 61.6 |
We report these results as they are; they support the diagnosis on the pixel-space model and they do not support it on the latent model.
ImageNet-64 EDM (pixel space): the mechanism reproduces, the numerical band does not transfer. On the noise family the ordering exact < uniform-t < band holds (18.4 < 28.4 < 32.5 < 40.2), and the four lowest-noise buckets classify at 9%, 7%, 7% and 9%, at or below the 10% chance floor, while buckets 8–11 classify at 71–79%. However, the contaminated region is wider than on CIFAR-10 (it reaches bucket 4 at severity 3 and bucket 6 at severity 5, against buckets 0–2 on CIFAR-10), so the bands locked on CIFAR-10 recover only a small part of the gap; a band whose lower edge is selected on the other corruption types (leave-one-corruption-out) reaches 83.8. We have not tested the cause; the corruption constants at 64×64 are stronger than CIFAR-10-C (Gaussian σ of 0.12 and 0.18 against 0.08 and 0.10), and the same log-SNR leaves more recoverable signal at higher resolution. The trade-off the paper reports for fog and contrast is also sharper here: those corruptions are best classified at the clean end (contrast, severity 5: exact 69.6, uniform-t 23.4, band 2–15 8.6), the exact score beats uniform-t on 6 of the 14 settings (contrast and fog at both severities, defocus blur and impulse noise at severity 5), and any band that repairs the noise family costs about 15 points on the non-noise settings. Over all 14 settings the locked band gives no net gain over uniform-t (51.9 vs 52.8).
SiT-XL/2 (latent space): the noise-family result does not reproduce. The order is reversed on all six noise settings (exact 59.5 > uniform-t 54.6 > band 52.7). After the VAE encoder, pixel noise damages the high-noise buckets most (buckets 12–15 classify at 13–26%) and the best buckets are low-noise ones (buckets 2–4, 56–58%). The exact score is still below uniform-t on clean data (95.4 vs 97.4) and on the non-noise settings (64.9 vs 69.9), because the single lowest-noise bucket, which receives 20% of the ITD weight, is the weakest low-noise bucket in every group (55% on clean data); but exact is below uniform-t on only 7 of the 14 shifted settings, the three scores are within sampling error over all 14 (62.6, 63.3, 62.9; the standard error of one setting is about 2 points), and dropping the clean end helps on no group. Our band rule should therefore not be applied to this model.
Response to Q1 of Reviewer WzfU (3/3)
(Continued, part 3 of 3.)
Stable Diffusion v1.4 (text-to-image, latent space): no low-noise failure and no gain from the band. We use the model zero-shot with the prompts "a photo of a [class name]" on the same 10 classes (500 images, 32 timesteps; clean data plus 8 corrupted settings: Gaussian, shot and impulse noise at severities 3 and 5, fog and contrast at severity 5). The exact score and uniform-t are within sampling error on every group (clean 95.4 vs 94.8; noise 78.0 vs 77.3; fog and contrast 84.0 vs 82.9; the standard error of one setting is about 2 points). Removing the low-noise end does not help: cutting at the two log-SNR edges of the paper gives 77.4 and 76.7 on the noise settings and 81.8 and 77.8 on fog and contrast (contrast alone: 81.6 exact, 70.2 with the wider cut). As with SiT, pixel noise damages the high-noise timesteps most after the VAE encoder: at severity 5 the timesteps with log-SNR below −4.5 classify at 15–17%, against 44–47% for log-SNR between 3.1 and 0.8. One caveat limits this test: the noise schedule of this model ends at log-SNR 4.25, so only 1 of the 32 timesteps lies above the paper's edge at 3.11, and it carries 6.2% of the ITD weight. That timestep is the weakest of the low-noise ones (33% at severity 5), but it has too little weight to separate the scores (per-setting numbers: Table S3 of the supplementary PDF). Our band rule should not be applied to this model either.
Scope. The evidence is now: two 32×32 class-conditional models on all 97 CIFAR-10-C settings (N = 2,000 each), two ImageNet models on a 10-class, 500-image subset with 14 corrupted settings each, and Stable Diffusion v1.4 on the same subset with 8 corrupted settings (single evaluation, no repeated seeds). The full 1000-way ImageNet-C protocol was beyond the compute we had during the discussion period. In the revision we will (i) restrict the claim "likelihood weighting is anti-robust under noise-like shift" to pixel-space diffusion classifiers, (ii) report the SiT and Stable Diffusion results as negative results for latent models, and (iii) state that the band has to be re-selected per model and resolution; the band numbers in the paper are specific to 32×32.
We will add this experiment as a new subsection of Sec. 5, with full per-corruption, per-severity and per-bucket tables in a new appendix, and move our own U-Net to a secondary role in the revision.
Response to Q1 of Reviewer WzfU (1/3)
Thank you for the careful and constructive review! Both of your suggestions could be tested during the discussion period. One clarification on your closing remark in Weakness 1: we agree that part of what the paper documents is a limitation of diffusion classifiers as such, and the paper does not argue otherwise (ERM is stronger on Rotated MNIST, AugMix on CIFAR-10-C). Its claim is narrower: which part of the score causes the collapse, and that this is a property of the weighting rather than of one model, which is what your Q1 asks us to test.
Q1. Does the low-noise failure exist in larger or pretrained models (text-to-image, SiT)?
Yes in pixel-space models, no in the two latent models we tested. (1) With the pretrained class-conditional CIFAR-10 EDM of Karras et al. (2022), every finding of the paper reproduces on all 97 settings. (2) With the pretrained ImageNet-64 EDM (296M parameters), the low-noise failure under noise corruptions reproduces, but the band selected on CIFAR-10 is too narrow and has to be re-selected. (3) With SiT-XL/2 (675M parameters, latent space), the failure under noise corruptions does not appear; the ordering is reversed there. (4) With Stable Diffusion v1.4 (text-to-image, latent space), the exact score and uniform-t are within sampling error and the band gives no gain. Details follow; we report the negative results as they are and will narrow the paper's claim accordingly.
Setup. We use the authors' released
edm-cifar10-32x32-cond-vpcheckpoint (DDPM++ backbone, 56M parameters, σ-parameterised, trained by its authors with their own log-normal σ sampling and augmentation; our paper's U-Net has 13.3M parameters and was trained by us with the cosine schedule). Nothing else in the pipeline changes: the same 16 log-SNR buckets, the same shared (t, ε) pairs, the same ITD quadrature weights, the same band-selection rules, and the same 2,000 test images per setting (the first 2,000 of the paper's fixed permutation). EDM is queried at σ(t) = exp(−α(t)/2) with ε̂ = (x_σ − D(x_σ))/σ. We evaluated all 97 settings of the paper (15 corruptions × 5 severities, the 4 hold-out corruptions × 5 severities, clean, CIFAR-10.1). The "U-Net" rows below are the paper's model scored on the identical 2,000 images, so the two denoisers are compared on the same inputs.What reproduces.
EDM is the stronger model throughout (uniform-t 82.2 vs 74.1 for our U-Net on the same images), and its band-restricted classifier reaches 85.3 on the 15 corruptions, matching the AugMix reference in Table 3. At 32×32 the low-noise failure is therefore a property of what the diffusion likelihood weights, not of our architecture, our training, or the ε-parameterisation.