- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper aims to explore the utility of training samples. Specifically, whether this utility is an inherent property of the data itself or dependent on the model. The paper uses the crossover point between simple samples and coverage-based samples as a metric. They verify assumptions by keeping the selected subset fixed while varying the downstream learner. Experiments demonstrate that model width, input resolution, and the dimensions of feature maps can shift the range of budget allocations for which one strategy outperforms the other. However, these effects are highly conditional; For instance, the influence of model width on the crossover point diminishes when the information content is richer. Furthermore, within the ViT-Tiny configurations and budget constraints, the coverage-based strategy consistently shows superiority. Based on these experimental results, the data selection strategy must be evaluated for the target model and training protocol, rather than inferred only from parameter numbers. This paper mainly focuses on empirical study and does not propose a new method.
Critical strengths
-
I think the research problem is interesting. By fixing the selected subset across key comparisons, this paper effectively decouples the impact of the selection strategy from the variations in the downstream learner. The experiments cover different model widths, input resolution, and step size, providing a coherent line of reasoning.
-
The paper is easy to follow.
Critical weaknesses
-
My primary concern is about the practical implications of this work. While it deepens our understanding of data utility, it offers no guidance or specific methodology for how to actually select data in practice. It remains unclear how a practitioner should determine which strategy is better suited for their target learner, especially with limited resources. According to the paper, if applied directly to real-world decision-making, the proposed crossover-point metric requires training models on both types of subsets across multiple budget levels, which could incur high additional costs. It would be beneficial if the authors could more clearly highlight the practical utility of their findings.
-
Additionally, the core findings are based on a low-resolution (32*32) version of the ImageNet-100 dataset and models on the scale of ResNet-18. At the more information-rich 64x64 resolution, the width effect essentially disappears. Even on the full-resolution ImageNet-1k dataset, increasing the number of parameters results in only minor changes to the selection outcomes for each class. This raises questions about whether the observed phenomenon still holds any practical significance in real-world, large-scale data and model scenarios.
-
Figures 1 and 3 should be provided in a vector graphics format.
Questions for authors
Please see the weakness.