- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper aims to discuss the method of hard-example selection, where simply based on highest loss if not enough, and the authors provide some studies over Gaussian distribution and real dataset. The goal is to better understand why the random/uniform selection still be the competitive baseline or even outperform those data selection methods.
Critical strengths
None
Critical weaknesses
-
The clarity of the paper is unclear, e.g., what is the motivation of data selection, and how it affects the possible downstream applications, e.g. data selection during active learning.
-
The presentation needs significant improvement, e.g. the paper presentation is lack of clarity, causing hard to follow. In abstract, no mentioning what is the purpose to select data at all; or in the introduction, what is the p* function? and what is phi function at line 49?
-
Following point 2: In the section starting at line 154, it still does not clearly state what is the setup, it only mentioned what data is used but not used for what. many symbols are unclear, E.g., the defined rho is not referred but I guess it means 0.05 and 0.1 at line 158.
Questions for authors
The presentation flow needs to be significantly improved to better convey the paper's findings and novelty.
3 comments
Follow-up to Reviewer Sfjf
Dear Reviewer Sfjf,
The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.
In brief: the presentation has been rewritten following each of your comments. The abstract and the introduction now open from why data is selected, the abstract has no symbols, every symbol is defined where it first appears, and Setup states what each ingredient is used for.
If anything remains unclear or unresolved, please let us know and we will respond before the deadline.
Thank you again for your time.
Reminder to Reviewer Sfjf
Dear Reviewer Sfjf,
A brief reminder that the discussion period closes on Nov 18 (AoE), about a day from now.
The revised abstract and introduction are the quickest way to see the changes made in response to your comments. If they read clearly to you now, we would be grateful if you would reconsider your rating. We can still answer any remaining question before the deadline.
Your comments on presentation made the abstract and the introduction easier to read. We know this is a busy period, and we are grateful for your time.
Reply to Reviewer Sfjf
Thank you for the review. Your comments center on the presentation, and the presentation has been rewritten following each of them. We answer in the order of your review and state what changed.
Motivation and downstream use. The abstract and the introduction now start from why data is selected. Training on a small subset of the training set saves computation. A common rule keeps the examples that a briefly trained network finds hardest, on the view that they are the most informative. When only a small fraction is kept this rule fails: with 5% of CIFAR-10 kept, a ResNet-18 trained on the hardest images reaches 14.1% test accuracy against 62.0% for a random subset. The paper asks why. The abstract no longer contains symbols. Active learning is a different problem, since it chooses which unlabeled examples to label, whereas we select from a labeled training set. Related work now says so in one sentence, so that readers do not expect results on it.
Undefined symbols in the introduction. The introduction now states the threshold in words, and every symbol in the main text is defined where it first appears. Φ is the standard normal distribution function. p*(r) is the threshold retained fraction below which, in the two-class Gaussian model, the two selected class means exchange positions. It is set only by how far apart the two classes are, measured by r = μ/σ. Formally p*(r) = Φ(z*), where z* is the unique solution of λ(z) = r and λ(z) = φ(z)/Φ(z) is the inverse Mills ratio, with φ the standard normal density.
Setup section (line 154 onward). Setup now comes directly before the experiments and says what each ingredient is used for, in three headed paragraphs. Subsets: a scoring probe scores every training image, the EL2N subset keeps the highest-scoring images of each class, and the random subset is drawn uniformly. Training: a ResNet-18 is trained from scratch on each subset and judged by its final test accuracy. Class means: the two quantities measured later, the count of selected class means that cross the midpoint and nearest-mean accuracy, are defined here before they are used. Your reading of the fraction symbol is correct: it is the retained fraction, with values 0.05 and 0.1. It is now written f throughout the main text, and Setup states how it relates to the p of the Gaussian model in Section 3 (the two are equal for the EL2N subset). The Experiments section opens with a short paragraph that says what each subsection does.
What the paper finds and what is new. In each class, almost all images kept by the hardest-example rule are ones that the scoring probe assigns to other classes (about 96%, against about 49% of a random subset), so the mean of their features lies toward those classes. Of the 30 selected class means (10 classes, 3 seeds), 22 are closer to the nearest competing class than to their own class, and none are for the random subset. Three ways of changing which images are kept all raise accuracy, and the position of the class means orders the repaired subsets almost as their test accuracy does. The same failure appears on four datasets and four architectures. The failure itself had been reported. Paul et al. saw accuracy drop sharply at extreme pruning rates, and Zheng et al. found that methods keeping the highest-scoring examples fall below random selection at high pruning rates, which they attributed to worse coverage of the data. What is new is that the paper traces the failure to the position of the class means of the kept images, measures that position directly, and gives a two-class Gaussian model in which the threshold fraction can be computed.
Revision. In summary, the abstract and the introduction open from the motivation of saving computation, the abstract has no symbols, related work has the sentence on active learning, each symbol is defined where it first appears, Setup is split into Subsets, Training and Class means with a single symbol f for the retained fraction, and Experiments opens with a roadmap. The presentation has changed substantially, and we would be grateful if you would read the revised PDF and reconsider your rating.