- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether score-based data pruning disproportionately harms some classes because it keeps different numbers of examples per class, chooses hard examples within each class, or simply produces a less accurate model. It audits common comparisons by progressively controlling these factors, including a hard-versus-random comparison with identical per-class sample counts and an approximate accuracy match obtained by shortening one model's training schedule. On CIFAR-100, the usual same-budget comparison against random pruning is badly confounded by class-count imbalance; after fixing those counts, selecting harder examples still concentrates damage in under-retained classes. However, this result only survives an architecture change on CIFAR-100: it is weak on Food-101, absent under noisy CIFAR-100 labels, and reverses on Tiny-ImageNet. The most credible contribution is therefore a practical audit protocol and a warning about controls and selection randomness, rather than a general finding that hard-example pruning harms classes.
Critical strengths
-
A useful control design for separating sources of class-level damage. The -mixture holds per-class counts exactly fixed while varying the hard-versus-random composition within each class. Together with the approximate accuracy-matched control, it distinguishes "how many examples per class", "which examples", and overall accuracy more carefully than a same-budget Random baseline. This is the paper's clearest reusable contribution.
-
Transparency about negative results and boundaries. The paper openly reports an abandoned aggregate hypothesis, a null result on CIFAR-100-N, a sign reversal on Tiny-ImageNet, and a borderline Food-101 result. Its reporting of score ties and selection-seed instability is also useful for reproducibility.
Critical weaknesses
-
The portable contribution is useful but limited in methodological novelty. The paper explicitly positions the four pitfalls and the control ladder as its main portable contribution. The concrete count-matched -mixture and the quantification of tie-induced selection variability are useful, but several broader lessons—using an external baseline, validating matched controls, and replicating all relevant sources of randomness—are established experimental-design principles. The novelty therefore lies mainly in assembling and demonstrating these controls for data pruning, which limits the significance of the paper's central methodological claim.
-
The discovery-setting effect contains an unresolved numerical discrepancy. In the Rung 5 paragraph of Section 5.3 (p. 6), the paper reports and , but then states that their difference is ; the displayed coefficients instead differ by . The value also appears in the following "Close to parallel" paragraph as the fitted vertical gap at the high end of the accuracy overlap, while Table 3 repeats as the headline CIFAR-100 result. The authors should clarify whether is a separately pooled or bootstrapped estimate rather than the difference of the displayed coefficients, and state exactly how it is computed.
-
The paper is unnecessarily difficult to follow. Rungs 1–2 are introduced mainly inside Figure 1, after which the prose in Section 4 begins directly with Rung 3; Section 5 then returns to Rung 1 and proceeds forward. The main paper also introduces local checkpoint terms such as "candidate anchor," "accuracy-matched anchor," and "screening/QC window." Although these terms are defined, later references often rely on the reader remembering both the rung number and the associated checkpoint construction. A more linear presentation that restates what each comparison holds fixed would make the otherwise straightforward experimental logic substantially easier to follow.
Questions for authors
The within-class effect changes substantially across CIFAR-100, Food-101, and Tiny-ImageNet. Since the utility of hard-example selection may depend on whether the selected training examples cover the difficult regions represented in the test distribution, have you estimated test-example difficulty using an independent model or ensemble and examined the effect separately on independently identified easy, medium-difficulty, and hard test examples? Such analyses could help distinguish a dataset-specific pruning mechanism from differences in difficulty composition and train–test alignment.
Dear authors,
Thank you for the constructive response. The narrower scope addresses several of my concerns.
Two issues remain unresolved:
Although the response improves my overall assessment, the main numerical results remain unverified, and the evidence does not yet establish the narrower contribution clearly enough to support acceptance. Addressing these issues, alongside the changes already proposed, would amount to a substantial revision whose soundness and significance would benefit from a new round of review. I therefore retain my current recommendation.