- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Diffusion Soft Labeling (DiffSL), a training-time calibration method that uses a class-conditional diffusion classifier to produce soft labels for original training samples and for inter-class augmented samples. Two techniques are introduced: top- class clipping, which restricts diffusion-classifier evaluation to a candidate set suggested by a pretrained reference classifier, and label correction, which zeroes out incorrect-class probabilities and re-normalizes among the correct class set. The classifier is trained with KL divergence to these soft labels. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet with ResNet-50/101 report the lowest ECE/AECE among the compared baselines while maintaining competitive accuracy.
Critical strengths
-
The core idea is clean and the empirical results are consistent. The story is intuitive: if soft labels approximate , they should calibrate better than one-hot labels, and a class-conditional diffusion classifier is a natural source. The ECE/AECE gains are consistent across six dataset–architecture settings, and the soft-label-source ablation (pretrained classifier vs. DDPM vs. EDM) is a useful sanity check.
-
The method is reasonably well engineered. Top- clipping makes diffusion-classifier labeling tractable at , and label correction addresses an obvious failure mode. The comparison with the diffusion classifier itself and with CSM re-annotation helps position the contribution.
Critical weaknesses
-
The computational cost and scalability are not analyzed clearly enough. DiffSL requires a class-conditional diffusion model and, for each training sample, diffusion-classifier evaluations. The paper reports NFE reduction relative to full-class DC, but not absolute wall-clock time, total GPU hours, or a comparison with the training cost of the baselines. This is the main reason I am not yet convinced the method is practically attractive: temperature scaling, label smoothing, and focal loss are far cheaper. I am not asking for ImageNet-1K experiments—I understand that is out of scope—but a serious cost–benefit analysis at the reported scale would materially change my view.
-
The quality of the diffusion-classifier soft labels is assumed rather than validated, and the augmentation component is not cleanly isolated. Section 4.1 gives the standard argument that soft labels help if they approximate , but there is no analysis of whether the diffusion classifier actually provides reasonable posterior estimates. Figure 3 shows a temperature-scaled DC, not raw DC calibration. If DC has systematic biases, DiffSL may simply transfer them. In addition, Table 5 shows that inter-class augmentation is not uniformly beneficial (CIFAR-10 prefers , CIFAR-100 prefers ), so the contribution of the core soft-labeling component versus the augmentation component is not cleanly separated.
Questions for authors
-
Can you report GPU hours for (a) diffusion-model training, (b) soft-label generation, and (c) final classifier training, and compare these with the baselines? A table with wall-clock time per 1k training samples for DiffSL vs. label smoothing / focal loss / temperature scaling would be helpful. A short discussion of how cost scales with and would also be useful for readers considering larger datasets.
-
Can you report raw (un-temperature-scaled) DC calibration, and compare DC soft labels with a simple proxy posterior (e.g., a small ensemble or a held-out estimate)? A sensitivity check to DC checkpoint quality (e.g., two checkpoints) would also be informative. Even a qualitative analysis—e.g., how often the DC top-1 disagrees with the true label, and whether label correction is doing most of the work—would help.
-
Can you report soft labeling only () across all six settings in the main table, not just in the ablation? This would let readers see whether the calibration gain comes primarily from soft labeling or from augmentation.
-
Can you clarify the validation-set size and tuning protocol for temperature scaling on the baselines, and confirm that the comparison is fair? A compute-matched CSM baseline using the same diffusion model would strengthen the comparison.
-
The top- clipping definition around Page 6 appears incomplete or truncated. Can you provide the full construction of and clarify how label correction interacts with the candidate set , especially for augmented samples where but the DC top-1 may be a third class? This is a presentation issue, but it currently makes the method harder to reproduce.
-
A recent AAAI 2025 paper, “Training Deep Neural Networks with Virtual Smoothing Classes” (Zhou et al.), proposes a rule-based soft label that addresses label smoothing’s side effects by introducing virtual classes. Could you discuss the conceptual relationship between DiffSL’s label correction and the virtual smoothing mechanism? When might data-driven soft labels from a generative model be preferable to well-designed rule-based soft labels?
-
Can you report reliability diagrams for DiffSL itself (not only for the temperature-scaled DC), and add Brier score / NLL? These are cheap to compute and would make the calibration claim more complete.
4 comments
Response to Reviewer yupu (2/4)
W2/Q2
(1) Raw DC calibration. DC approximates class-conditional log-likelihoods using the diffusion loss and constructs class probabilities from these estimates. Though the diffusion loss is derived from an evidence lower bound (ELBO), specific training configuration (e.g., viewing as constant 1 during training) may make this loss no longer an accurate estimation of ELBO. Accordingly, as described in Section 4.3, DiffSL incorporates temperature scaling into soft-label construction, selecting the temperature that minimizes calibration error to improve confidence calibration while preserving class rankings. We additionally report DC results before temperature scaling to show the calibration improvement achieved by this step.
| Dataset | Raw DC ECE (%) | Temperature-scaled DC ECE (%) |
|---|---|---|
| CIFAR-10 | 0.6207 | 0.0847 |
| CIFAR-100 | 3.0464 | 0.4623 |
| Tiny-ImageNet | 16.9801 | 0.8259 |
We compare DC and a pretrained ResNet-56 discriminative classifier after temperature scaling. Both use the same CIFAR test splits, fitting temperature by NLL on 2,000 images and evaluating on the remaining 8,000.
| Dataset | Model | ACC (%) ↑ | ECE (%) ↓ | NLL ↓ | Brier ↓ |
|---|---|---|---|---|---|
| CIFAR-10 | ResNet-56 | 94.51 | 0.9735 | 0.1871 | 0.0842 |
| CIFAR-10 | DC | 93.91 | 0.5945 | 0.1776 | 0.0884 |
| CIFAR-100 | ResNet-56 | 72.46 | 2.8222 | 1.0536 | 0.3846 |
| CIFAR-100 | DC | 80.54 | 2.6940 | 0.7644 | 0.2794 |
DC achieves lower ECE and NLL on both test sets. Its Brier score is lower on CIFAR-100 but slightly higher than ResNet-56 on CIFAR-10. Low training ECEs and these test comparisons support using DC predictions to construct high-quality soft labels.
(2) Soft-label sources and diffusion model choice. In Table 2, both EDM and DDPM soft labels improve calibration over pretrained-classifier labels under identical CIFAR-10/ResNet-50 settings using only original data. EDM maintains comparable accuracy. These results support the calibration benefit of diffusion soft labeling across model choices.
(3) Prediction disagreement and label correction. For original training samples, we apply correction only when the DC top-1 prediction within the candidate classes disagrees with the ground-truth label. The table reports correction rates for each reference classifier.
| Dataset | ResNet-50 correction rate (%) | ResNet-101 correction rate (%) |
|---|---|---|
| CIFAR-10 | 1.822 | 1.822 |
| CIFAR-100 | 8.032 | 8.032 |
| Tiny-ImageNet | 11.898 | 12.203 |
87.797%–98.178% of the original training samples retain their soft labels, so DC probabilities still provide soft supervision for most samples. We use ground-truth one-hot labels to correct only the minority with incorrect predictions, ensuring consistency with known labels.
Table 3 supports label correction for improving calibration and accuracy while preserving soft supervision for most samples.
W2/Q3
Thank you for this suggestion. We will consider adding a row with these results to the main results table in the revised version. Note that Table 5 of the current manuscript already reports ECE results for soft labeling without augmentation () across all six settings.
Q4
(1) Validation splits and temperature tuning. As described in Appendix “Experiment Details,” we reserve 5,000 training images for validation on each CIFAR dataset and 10,000 on Tiny-ImageNet, leaving 45,000 and 90,000 for classifier training. We determine the splits using each run's random seed. Model selection and temperature fitting use the validation set, and we fix the selected settings before test evaluation. We select classifiers by validation accuracy and search to minimize validation ECE with 15 equal-width bins. All baselines use the same protocol, and Table 1 reports results before and after temperature scaling.
(2) Diffusion model used by CSM. CSM and DiffSL already use the same pretrained EDM weights for image generation on all three datasets. In Table 8, we compare the labeling methods on the same augmented images. DiffSL achieves higher accuracy and lower ECE and AECE than CSM re-annotation.
Response to Reviewer yupu (3/4)
Q5
We have rechecked Section 4.3 and confirmed that the construction of and the label correction rule are complete.
For an augmented image mixing classes and with , let . We form by adding the classes outside ranked highest by reference classifier . We renormalize the calibrated DC probabilities over the candidate classes and set all other probabilities to zero. If the DC top-1 prediction lies outside , correction retains and renormalizes only the probabilities within , preserving their ratio.
For example, let and . Suppose the normalized DC probabilities for are . Correction produces (0.6, 0.4, 0), even though c has the highest probability. The resulting label remains soft and preserves the relative probabilities of the two generating classes.
We will consider adding this example to the revised version to make the labeling procedure easier to follow.
Q6
Virtual Smoothing (VS; Zhou et al., 2025) and our label correction both use known labels to avoid conflicting supervision. VS assigns smoothing probability to virtual classes and keeps the targets for incorrect real classes at zero, thereby regularizing confidence. DiffSL estimates probabilities over real classes using DC and applies correction only when the predicted class conflicts with the known label set. When the prediction is consistent, the full soft label is retained, including probabilities assigned to other real classes. Our correction therefore allows us to preserve estimated label uncertainty while maintaining consistency with known labels.
VS assigns equal target probabilities to all virtual classes, irrespective of the sample. DiffSL instead estimates a soft label for each image using the learned class-conditional distributions. This data-driven approach aims to better approximate the optimal soft target in Section 4.1 by capturing both the uncertainty of each sample and the relative probabilities of competing classes.
Response to Reviewer yupu (4/4)
Q7
DC temperature scaling is part of our soft-label construction in Section 4.3. DiffSL classifier predictions use no additional post-hoc temperature scaling. Q2 reports both raw and temperature scaled DC results.
We supplement all methods and all six settings in Table 1 with Brier score and NLL, using the same checkpoints and ten seeds. Parentheses show baseline results after temperature scaling with validation-selected temperatures. The best value for each setting and metric is bold.
Brier score ↓
| Method | CIFAR-10 (ResNet-50) | CIFAR-10 (ResNet-101) | CIFAR-100 (ResNet-50) | CIFAR-100 (ResNet-101) | Tiny-ImageNet (ResNet-50) | Tiny-ImageNet (ResNet-101) |
|---|---|---|---|---|---|---|
| CE | 0.0789 (0.0733) | 0.0810 (0.0746) | 0.3306 (0.3051) | 0.3396 (0.3073) | 0.4662 (0.4648) | 0.4568 (0.4545) |
| MMCE | 0.0789 (0.0733) | 0.0813 (0.0745) | 0.3345 (0.3081) | 0.3415 (0.3072) | 0.4641 (0.4626) | 0.4589 (0.4556) |
| LS | 0.0803 (0.0810) | 0.0821 (0.0829) | 0.3148 (0.3110) | 0.3361 (0.3280) | 0.4662 (0.4659) | 0.4553 (0.4553) |
| Mixup | 0.0753 (0.0755) | 0.0725 (0.0734) | 0.3069 (0.2981) | 0.3028 (0.2907) | 0.4577 (0.4572) | 0.4476 (0.4472) |
| FL | 0.0792 (0.0776) | 0.0790 (0.0779) | 0.3013 (0.2989) | 0.2987 (0.2973) | 0.4995 (0.4949) | 0.4874 (0.4837) |
| FLSD | 0.0796 (0.0778) | 0.0789 (0.0780) | 0.3064 (0.3041) | 0.3016 (0.3006) | 0.5015 (0.4968) | 0.4901 (0.4863) |
| MbLS | 0.0828 (0.0828) | 0.0825 (0.0825) | 0.3175 (0.3175) | 0.3323 (0.3323) | 0.4693 (0.4692) | 0.4576 (0.4576) |
| CALS | 0.0819 (0.0768) | 0.0827 (0.0772) | 0.3263 (0.3051) | 0.3421 (0.3127) | 0.4685 (0.4685) | 0.4579 (0.4579) |
| RankMixup | 0.0899 (0.0857) | 0.0920 (0.0889) | 0.3433 (0.3234) | 0.3464 (0.3233) | 0.5190 (0.5190) | 0.5071 (0.5072) |
| SKMixup | 0.0785 (0.0769) | 0.0766 (0.0751) | 0.3089 (0.3079) | 0.2944 (0.2942) | 0.5204 (0.5060) | 0.5018 (0.4886) |
| CSM | 0.0629 (0.0630) | 0.0622 (0.0623) | 0.2947 (0.2947) | 0.2942 (0.2938) | 0.4601 (0.4593) | 0.4486 (0.4467) |
| DiffSL | 0.0718 | 0.0721 | 0.2720 | 0.2691 | 0.4301 | 0.4679 |
NLL↓
| Method | CIFAR-10 (ResNet-50) | CIFAR-10 (ResNet-101) | CIFAR-100 (ResNet-50) | CIFAR-100 (ResNet-101) | Tiny-ImageNet (ResNet-50) | Tiny-ImageNet (ResNet-101) |
|---|---|---|---|---|---|---|
| CE | 0.2034 (0.1623) | 0.2205 (0.1686) | 0.9751 (0.8901) | 1.0249 (0.8983) | 1.4129 (1.4047) | 1.3789 (1.3666) |
| MMCE | 0.2041 (0.1628) | 0.2282 (0.1715) | 0.9905 (0.9015) | 1.0323 (0.9072) | 1.4081 (1.3977) | 1.3899 (1.3726) |
| LS | 0.2198 (0.2142) | 0.2247 (0.2206) | 0.9718 (0.9799) | 1.0484 (1.0360) | 1.4303 (1.4278) | 1.3942 (1.3942) |
| Mixup | 0.1767 (0.1763) | 0.1769 (0.1736) | 0.8348 (0.8313) | 0.8242 (0.8129) | 1.3842 (1.3817) | 1.3448 (1.3422) |
| FL | 0.1895 (0.1649) | 0.1870 (0.1655) | 0.8053 (0.7774) | 0.8036 (0.7772) | 1.4907 (1.4821) | 1.4450 (1.4382) |
| FLSD | 0.1908 (0.1658) | 0.1877 (0.1671) | 0.8173 (0.7900) | 0.8097 (0.7852) | 1.4935 (1.4864) | 1.4536 (1.4481) |
| MbLS | 0.2200 (0.2200) | 0.2198 (0.2198) | 1.0226 (1.0226) | 1.0709 (1.0709) | 1.4350 (1.4346) | 1.3914 (1.3914) |
| CALS | 0.2174 (0.1797) | 0.2255 (0.1822) | 0.9733 (0.9087) | 1.0456 (0.9448) | 1.4249 (1.4249) | 1.3896 (1.3896) |
| RankMixup | 0.2149 (0.1833) | 0.2195 (0.1975) | 0.9821 (0.8994) | 0.9956 (0.8986) | 1.5920 (1.5920) | 1.5490 (1.5492) |
| SKMixup | 0.1905 (0.1827) | 0.1886 (0.1817) | 0.8280 (0.8103) | 0.7912 (0.7812) | 1.6150 (1.5576) | 1.5500 (1.4981) |
| CSM | 0.1328 (0.1334) | 0.1346 (0.1349) | 0.8033 (0.8015) | 0.7963 (0.7949) | 1.4018 (1.3978) | 1.3622 (1.3545) |
| DiffSL | 0.1470 | 0.1485 | 0.6942 | 0.6841 | 1.2974 | 1.4116 |
DiffSL achieves the lowest ECE across all six settings, together with the lowest NLL and Brier in the three settings where it also achieves the highest accuracy. Note that NLL and Brier scores are not pure calibration metrics, they reflect a mixture of discrimination and calibration ability. On CIFAR-10 and Tiny-ImageNet with ResNet-101, the lower NLL and Brier of some baselines accompany their higher accuracy, while DiffSL retains its advantage in confidence calibration.
Response to Reviewer yupu (1/4)
We thank the reviewer for the detailed feedback and for recognizing our clear core idea and consistent empirical results.
We reuse RDC's pretrained EDM (Chen et al., 2024) without additional training. We can release datasets with soft labels so users can train their own models directly on them, with only a minor change to the loss function and no other changes to the training code. Even in a cold-start setting where all models must be trained from scratch, labeling is required only once per dataset and labeling configuration. These labels can be reused throughout repeated model training, hyperparameter tuning, and evaluation without relabeling.
The table below estimates augmented data generation(w/ labeling) costs in GPU-hours on NVIDIA RTX 4090. Labeling uses selected temperatures and top-15 on CIFAR-100/Tiny-ImageNet. A dash means no augmentation.
Per 1,000 original images, CIFAR-10 DC scoring takes 0.2837 GPU-hours. Top-15 labeling takes 0.5218 / 0.5177 GPU-hours on CIFAR-100 and 0.6786 / 0.6763 on Tiny-ImageNet for ResNet-50 / ResNet-101.
The table below reports estimated GPU-hours per training run. Parentheses indicate the number of augmented images used per original image.
Without generated augmentation, training costs are comparable to the baselines. Augmentation increases training cost but reduces CIFAR-100/ResNet-50 ECE from 5.16% to 1.08% in Table 4. Classifier training and inference require no diffusion computation.
DC scoring scales as O(NlabelNϵk) for Nlabel images, Nϵ noise samples per class score, and k candidate classes per image. Relative to all C classes, k=15 reduces class evaluations by 85% on CIFAR-100 and 92.5% on Tiny-ImageNet. With a fixed number of generated images per class pair, the pool scales as O(C2). For large C, one can use inter-class distances to form pairs only between nearby classes, which would reduce the pool size to O(C) by keeping a fixed number of neighbors per class.