- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The manuscript proposes PixCalibrate, a training-time pixel-level mixing augmentation method. Its core idea is to use AUM-guided easy-anchor/hard-source pixel-level mixing to simulate fine-grained, spatially distributed corruptions, and to combine it with a confidence penalty to simultaneously improve accuracy and calibration under corruption. The labels are mixed in proportion to the replacement fraction. The authors evaluate accuracy and calibration on several corruption benchmarks, additional shifted test sets, and ImageNet variants. The experiments are reasonably extensive, but the novelty looks limited: the method is essentially a combination of AUM, pixel mixing, and an entropy penalty.
Critical strengths
- The method is simple and relatively inexpensive.
- The authors report corruption-wise results and an analysis excluding pixel-level corruption types.
- The experimental coverage is reasonably extensive, with evaluation on CIFAR-10.1, CIFAR-10.2,CINIC-10, and CIFAR-10-W.
- Tccuracy is highest among the compared-to methods in each case, albeit usually by a small margin? That said, the comparison is restricted to Mixup and CutMix.
Critical weaknesses
- The method is mainly a combination of AUM, pixel-level mixing, and a confidence penalty, and is closely related to CalibrateMix, Mixup, CutMix, etc.; its theoretical novelty is limited.
- The sampling procedure of the mask is not sufficiently specified.
- AUM splits easy/hard samples by the median within each batch. However, when the batch is small, classes are imbalanced, or label noise is high, the split may be unstable.
- The comparison with CalibrateMix may not be entirely fair, since the original CalibrateMix setting is semi-supervised, whereas this paper is fully supervised.
- The mechanism is not isolated. Replacing 40% of randomly located pixels with values from another image creates a noise-like perturbation. It seems that the largest CIFAR-10-C improvements occur for Gaussian, shot, and impulse noise, whereas brightness barely changes. The same-class and different-class versions are also close in accuracy. This is consistent with noise-like augmentation being important, although it does not prove that label mixing is unnecessary. A matched pixel-noise replacement control, with and without mixed labels, might be essential to distinguish the proposed mechanism from a simpler augmentation.
- The advantage of AUM-based pairing is not convincingly established. The CIFAR-10-C ECE improvement is worth noting, but most differences are small. Without seed-level paired uncertainty for this ablation, the evidence does not yet support a strong claim that pairing is a key ingredient.
- Relevant robustness and calibration baselines deserve more attention. The results on Table 55 do not seem to support a general claim of state-of-the-art calibration is not supported on this benchmark. AugMix and PixMix should be included in the main comparison under a clearly matched training recipe, alongside a simple noise baseline; temperature scaling and RegMixup would also be informative. Some existing baselines perform below the unaugmented model, which warrants an explanation of tuning and implementation. Worth mentioning is that published numbers from different WRN variants are not directly comparable.
- The ImageNet evidence is limited by the reference training recipe. The reported clean ResNet-50 baseline is 71.76%, and its ImageNet-C accuracy is 26.93%. These are weak relative to commonly used ResNet-50 training results, making it difficult to judge how much of the improvement would persist against stronger baselines. The ImageNet comparison also lacks AugMix/PixMix and seed variability.
- The statistical analysis requires more care. Testing across corruption–severity cells from the same trained models does not provide independent training replications. Seed-level results, ideally with paired model comparisons, would be a more appropriate basis for significance claims. The similarity analysis also lacks enough detail about corruption conditions and variability to interpret the very small frequency-distance differences.
Questions for authors
- What is the specific sampling distribution of the binary mask? Is it an independent per-pixel Bernoulli?
- Have you tried spatially correlated masks to better simulate corruptions with spatial structure, such as fog, weather, and blur?
- What proportion of the hard samples are mislabeled/ambiguous?
- How does the method compare with random pixel replacement, impulse noise, and same-image pixel shuffling at the same replacement fraction?
- Can the authors report seed-level uncertainty for the AUM-versus-random-pairing comparison?
- How do AugMix, PixMix, and RegMixup perform under exactly the same CIFAR and ImageNet training settings?
- Why is the ResNet-50 clean baseline at 71.76%, and what changes under a stronger conventional recipe?
5 comments
Author response to Reviewer CcY1
>>W2, Q1: binary mask
We thank the reviewer for pointing out the need for clarification. As described in Eq. 4, PixCalibrate generates a binary mask by uniformly sampling spatial pixel locations, with a fixed replacement fraction (). The same mask is applied across all RGB channels, ensuring that complete pixel values are replaced rather than individual color channels.
>>W3: unstable split on noisy and small data batch
We thank the reviewer for raising this concern. We acknowledge that batch-wise median splitting may introduce variability in easy/hard assignments. However, despite these potential variations, our experimental results demonstrate that the simple median-based difficulty split consistently outperforms existing baselines across the evaluated corruption benchmarks in both accuracy and calibration.
>>W4: CalibrateMix on fully supervised
We thank the reviewer for pointing out the need for clarification. We preserve CalibrateMix's original AUM-based difficulty estimation, top-k feature-dissimilarity selection, bidirectional pairing strategy, and global Mixup operation. Since all training samples are labeled, the original easy-labeled–hard-unlabeled and easy-unlabeled–hard-labeled pairings are adapted to easy-labeled–hard-labeled pairings in both directions, using ground-truth labels instead of pseudo-labels. The SSL-specific unsupervised objectives are omitted, while the remaining mixing loss is retained. All methods follow the same fully supervised training protocol. We will clarify these adaptation details in the revised manuscript.
>>W5, Q4: Random Noise instead of pixels
We thank the reviewer for this observation. Unlike Gaussian or salt-and-pepper noise, which introduce perturbations from predefined distributions, PixCalibrate generates image-dependent perturbations using pixels from another training sample. As illustrated in Fig. 2, this produces spatially distributed variations while retaining recognizable anchor content.
To further investigate whether the improvements can be attributed to simpler noise-like augmentations or label mixing, we conducted additional experiments on CIFAR-10-C and report the results below:
| Method | Cifar10c ece | Cifar10c acc |
|---|---|---|
| FullySupervised | 15.18 | 73.48 |
| FullySupervised + gaussian noise augmentation | 11.25 | 76.95 |
| Pixel shuffle | 8.93 | 83.52 |
| Random pixel values | 8.66 | 83.16 |
| Original label | 7.81 | 83.35 |
| PixCalibrate (ours) | 7.64 | 84.70 |
The results show that Gaussian noise augmentation improves upon the FullySupervised baseline, confirming that noise-like perturbations contribute to corruption robustness. Pixel shuffling and random pixel replacement provide further improvements, but both remain behind PixCalibrate in accuracy and calibration. Furthermore, using only the original anchor label while keeping pixel-sampling and replacement unchanged reduces accuracy from 84.70% to 83.35% and increases ECE from 7.64 to 7.81. These findings suggest that image-dependent pixel replacement is beneficial beyond simpler perturbations, while proportional soft-label mixing provides an additional improvement.
We acknowledge that the largest improvements occur on noise-like corruptions. However, the modest gain on brightness is partly explained by the already high baseline accuracy (94.13%), which PixCalibrate maintains at 94.22%. Importantly, the benefits extend beyond noise-like corruptions. Appendix A.6.1 shows that, after excluding five pixel-level corruption types, PixCalibrate achieves 86.80% accuracy and 5.44 ECE. These results support the broader effectiveness of PixCalibrate beyond noise-like perturbations.
>>W6: Benefit of AUM-pairing over random pairing
We thank the reviewer for this observation. We would like to clarify that AUM-guided pairing is a complementary component rather than the primary novelty of PixCalibrate, which lies in its pixel-level sampling and replacement mechanism. Nevertheless, AUM-guided pairing consistently improves calibration across all three seeds on CIFAR-10-C, reducing ECE from 8.48 ± 0.29 (random pairing) to 7.63 ± 0.73
Author response to Reviewer CcY1
>>W7, Q6: PixMix, AugMix, and temperature scaling
We thank the reviewer for this suggestion. First, we clarify that all corresponding experiments have been performed under the same training settings. Second, AugMix and PixMix are already included in Table 5 under the same training settings as PixCalibrate. We present them separately to distinguish robustness-oriented augmentation strategies from the conventional image-mixing baselines in Table 1. We will make this distinction and the reference to Table 5 more explicit in the revised manuscript.
To further strengthen the evaluation, we conducted additional experiments with post-hoc temperature scaling on CIFAR-10-C. For temperature scaling, we optimize a single temperature parameter for each trained model by minimizing negative log-likelihood on a validation set. The results are reported in the table below:
| Method | Cifar10c ece | Cifar10c acc |
|---|---|---|
| FullySupervised | 15.18 | 73.48 |
| FullySupervised + Temp Scaling | 6.70 | 73.48 |
| PixCalibrate + Temp Scaling | 4.89 | 84.70 |
| PixCalibrate | 7.64 | 84.70 |
The results show that, temperature scaling reduces the FullySupervised baseline's ECE from 15.18 to 6.70.In contrast, PixCalibrate achieves 84.70% accuracy, and applying temperature scaling further reduces its ECE from 7.64 to 4.89, outperforming the temperature-scaled FullySupervised baseline in both accuracy and calibration. These findings demonstrate that PixCalibrate's robustness improvements are complementary to post-hoc confidence calibration. We will update the manuscript with the added experiments.
>>W8, Q7: Imagenet training
We thank the reviewer for this observation. We clarify that our ImageNet experiments use the standard ResNet-50 architecture and follow the training protocol used in CalibrateMix on the FullySupervised setting on ImageNet.
Importantly, all methods use identical architectures, optimization settings, learning-rate schedules, and training durations, ensuring a controlled comparison. Our evaluation focuses on the relative improvements under matched training conditions. To further strengthen the ImageNet evaluation, we conducted additional experiments with CutMix and ResizeMix under the same training configuration and report the results below:
| Imagenet ece | Imagenet acc | Imagenet-c ece | Imagenet-c acc | |
|---|---|---|---|---|
| CutMix | 5.40 | 71.89 | 22.67 | 28.92 |
| ResizeMix | 5.19 | 72.15 | 21.18 | 29.61 |
| PixCalibrate (ours) | 3.74 | 72.79 | 15.86 | 32.53 |
As can be seen from the table above, PixCalibrate consistently outperforms both augmentation baselines in accuracy and calibration on clean and corrupted ImageNet. Compared with the stronger baseline, ResizeMix, PixCalibrate improves ImageNet-C accuracy by 2.92 points while reducing ECE by 5.32 points. These findings provide additional evidence that the benefits of PixCalibrate extend to large-scale image classification under matched training conditions.
>>W9: Statistical Significance and Frequency Distance.
We thank the reviewer for this observation. As per your suggestion, we performed statistical analysis using paired comparisons across three independent training seeds. The revised tests confirm that PixCalibrate's improvements over the FullySupervised baseline remain statistically significant (<0.05)), supporting the reliability of our reported accuracy and calibration gains. We will update the manuscript to reflect this corrected procedure.
Regarding the similarity analysis, we agree that the frequency-distance difference between PixCalibrate (0.1716) and Mixup (0.1717) is marginal and does not provide strong evidence of superiority. However, the more notable result is PixCalibrate's higher SSIM (0.36), compared with Mixup (0.20) and CutMix (0.27) which provides evidence that PixCalibrate's augmentation perturbations exhibit greater structural similarity to the evaluated corruption perturbations.
>> Q2:
We thank the reviewer for the question. PixCalibrate uses randomly distributed pixel-level masks rather than spatially correlated masks, as our objective is to generate diverse, fine-grained perturbations without explicitly modeling specific corruption types. Nevertheless, our results show improvements on spatially structured corruptions, including defocus blur (83.44% to 87.56%), zoom blur (77.53% to 84.15%), snow (82.28% to 88.86%), and fog (86.57% to 89.25%) on CIFAR-10-C. Furthermore, Appendix A.6.1 demonstrates that the improvements persist after excluding five pixel-level corruption types. These findings suggest that independently sampled pixel masks can also improve robustness to broader corruption patterns.
Author response to Reviewer CcY1
We thank the reviewer for the valuable feedback. We address the weaknesses and questions below and we will incorporate them in the paper.
>>W1:
We thank the reviewer for raising this point. PixCalibrate methodological contribution centers on the design of a new augmentation technique, which naturally resembles test-time corrupted images. While CalibrateMix and random Mixup directly interpolate pixel values across two images, i.e., all contiguous pixels of an image are overlaid with all contiguous pixels from another image, according to the mixing ratio (see Fig. 2), PixCalibrate samples (uniformly at random) individual pixels from a hard image and places them into an anchor image (by completely replacing the anchor’s pixels with the pixels sampled from the hard image in exact same positions). More precisely, PixCalibrate uniformly samples individual pixel locations and replaces their values with those from a paired image, producing spatially distributed perturbations while retaining much of the anchor's structure (see also Fig. 2), especially because we preserve 60% of the anchor's pixels intact and replace only 40% with the pixels sampled from the hard image. Please note that, unlike CutMix, these 40% replacements are spatially scattered rather than concentrated in contiguous patches that create an entire patch of pixels being replaced. Figure 2 visually illustrates this distinction, showing that PixCalibrate introduces localized variations while retaining the anchor's overall appearance. Table 3 provides quantitative support: PixCalibrate achieves higher structural similarity to corruption perturbations (SSIM: 0.36) than Mixup (0.20) and CutMix (0.27).
In addition, we incorporate difficulty-aware pairing, which is inspired from CalibrateMix. However, while CalibrateMix combines easy labeled samples with hard pseudo-labeled samples, and hard labeled samples with easy pseudo-labeled samples, PixCalibrate uses a fixed direction to the construction: the easy sample provides the anchor, while the hard sample provides the perturbation. This design aims to preserve the features of the anchor while introducing challenging variations from the hard sample. As demonstrated in Table 5, reversing this direction by using hard samples as anchors can produce more ambiguous augmented samples, reducing label reliability. At our default replacement ratio (ρ=0.4), 60% of spatial locations retain the easy image’s original pixel values, while the remaining locations receive pixels from the hard image. Thus, difficulty determines both which samples are paired, which sample acts as anchor, and which sample acts as the perturbation source.
Importantly, these methodological differences contribute to improvements beyond those provided by confidence regularization. Although the confidence penalty complements our pixel-sampling strategy by encouraging higher predictive entropy, removing it still yields substantial improvements over CalibrateMix (as shown in the table below and Tables 1 and 5 in the paper):
Furthermore, Appendix A.6.2 applies the same confidence penalty to Mixup and CutMix; these variants still underperform PixCalibrate without the penalty on both accuracy and ECE. This supports treating confidence regularization as a complementary component rather than the primary explanation for the gains.
We will revise the presentation to distinguish the inherited ingredients from the proposed construction. Specifically, our contribution is the integration of difficulty-dependent anchor/perturbation roles with spatially distributed mixing for jointly improving robustness and calibration under corruption. The pixel-sampling and replacement operator is central to this contribution, and the role-selection ablation provides evidence for the additional design choice beyond mixing granularity.