- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Coreset selection and dataset distillation each have their own strengths and limitations. Hybrid Dataset Curation (HDC) therefore attempts to combine representative real samples with synthetic samples. Source-agnostic HDC approaches select from a combined pool of real and synthetic samples, which can result in too many real samples being selected. Fixed-ratio approaches instead use a preset hyperparameter to determine the proportion of the two types of samples, limiting the adaptability of the selected subset. This paper proposes Differentiated Hybrid Curation (DHC). DHC first builds a candidate pool with balanced numbers of real and synthetic samples using nearest-neighbor matching in CLIP feature space. It then defines a source-aware score, DIS, based on the difference in gradient norms between expert models trained separately on the two types of data. Finally, it greedily selects a subset by combining a gradient-based representativeness score with a diversity-aware penalty.
Critical strengths
-
The authors are transparent about the less favorable results on ImageNette and provide a detailed appendix.
-
IGD_NNr is an interesting byproduct of this work. At low IPC, it outperforms most existing coreset selection baselines in the reported experiments.
Critical weaknesses
1. Limited evaluation across IPC settings.
The scope of the evaluation is relatively limited. The authors only evaluate ImageNette and ImageWoof at different IPC settings. On ImageNet-1K, they evaluate the method only at IPC 10. This is not sufficient evidence to establish the effectiveness of the method across different IPC settings, since some methods perform well at low IPC but not at higher IPC. For example, IGD_NNr performs well at low IPC but sometimes fails to outperform Random at higher IPC. Therefore, the current results showing that DHC outperforms other methods at low IPC do not necessarily mean it will perform well at higher IPC. To make the evaluation on ImageNet-1K more convincing, I would suggest conducting more experiments at different IPC settings. Moreover, based on the current results, DHC fails to outperform CCFS in three of the four evaluated settings on ImageNette.
2. Potential limitations of DIS.
Since the two experts used to select samples are trained only on real or synthetic data, respectively, they may have lower gradient norms on their own training data. Meanwhile, evaluating samples from the other source is an out-of-distribution scenario, which may result in higher gradient norms. Therefore, I suspect that DIS mainly reflects which source the samples come from, rather than their individual utility. One way to investigate this is to train each expert on only a portion of its corresponding data and then compare the gradient norms of held-out samples from the same source with those of samples from the other source.
Questions for authors
Why do the authors consider DHC's selection strategy adaptive when its hyperparameters are tuned separately for each IPC setting? What is the practical advantage of this approach over tuning the real-to-synthetic ratio in CCFS?