- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates why models trained on small coresets tend to favor high-variance features over potentially more predictive low-variance features. It introduces Coreset Covariance Jitter (CCJ), which perturbs training images according to estimated within-class feature variances, and combines it with a QDA classifier at inference to recover complementary class information. The paper provides a theoretical analysis of feature availability and predictivity, supported by experiments on a controlled synthetic task. Extensive evaluations on CIFAR-100 and ImageNet-1K show consistent improvements across coreset selection methods at 10 images per class, with additional validation across model architectures, label-free settings, and dataset distillation. The method adds little computational or storage overhead, though its benefits are less consistent at higher coreset budgets.
Critical strengths
-
Strong empirical performance with minimal overhead. CCJ consistently improves a wide range of coreset selection methods at IPC-10 on CIFAR-100 and ImageNet-1K. Its low computational and storage costs make it a practical enhancement to existing pipelines rather than a replacement for them.
-
A useful theoretical perspective with a simple, generalizable method. The paper connects feature availability and predictivity to explain why models trained on small coresets favor high-variance features. CCJ translates this insight into a lightweight augmentation, while QDA provides complementary information at inference time. The combination is validated across multiple architectures and label-free settings, strengthening the practical relevance of the contribution.
Critical weaknesses
Dependence on full-dataset statistics limits the practical interpretation of coreset compression. Both CCJ and QDA require statistics computed from the full training set, including class-conditional information. Although the additional storage is small, the method does not eliminate the need to access the original dataset and may be unsuitable when data access or privacy constraints are central motivations for coreset selection. This weakens the claim that the method provides a broadly applicable solution to data reduction.
Questions for authors
- Since CCJ and QDA use statistics computed from the full training set, how sensitive are the results to the accuracy of these statistics, particularly when only a small subset of the data is available or class labels are noisy?
Response to Reviewer dXeM
Thanks for taking the time to review our submission. It is true that as you pointed out, CCJ and QDA use the full dataset statistics when the coreset is being selected from the full dataset. Sensitivity and data privacy is an interesting challenge, but that is mostly a consideration in the related field of dataset distillation. In coreset selection we always provide a non-anonymised coreset by definition, and the only way to account for sensitivity of data would be to remove it from the training set before computing the coreset. Our answer to Q1 below experimentally tests this setting. We further added two experiments to loosen the reliance on full dataset statistics (App J).
Q1. How sensitive are the results to the stored statistics when only a subset of the data is available or the labels are noisy?
The QDA uses a mean and a covariance per class from the training set, so it is more sensitive than CCJ, particularly at the extremes. With ten images per class, it loses 2.67 points (Tab 35). From 250 images per class onwards the loss is only 0.34 points. The QDA is also robust to moderate label noise: replacing 10% or 20% of the labels costs only 0.15 and 0.35 points, and only heavy noise costs more, 1.37 points at 60% (Tab 36).
CCJ uses the pooled within-class variance of each DCT coefficient, from the dataset. We introduced a new experiment to test the effects of using the variance calculated from a subset on CCJ. CCJ's performance gain drops only by 0.09 when we calculate the variance from 10% of the CIFAR-100 dataset. If we use the coreset's own images, its gain decreases by 0.5. CCJ is robust to noisy labels, and we observe that the noisy labels actually provide a small improvement (Tab 34) but this is within run-to-run variance. We have added a more detailed discussion of this in App J.