acceptodds
Under review as a conference paper at ICLR 2027

When Hard-Example Selection Fails: Gaussian Thresholds and Empirical Diagnostics

Abstract

Selecting the highest-loss examples can fall tens of accuracy points below random sampling at small retained fractions. We explain when this happens in a Gaussian model and test its predictions on image classification. For a fixed aligned score, we identify a threshold p*(r), given by one scalar equation in the signal-to-noise ratio r, and prove that with high probability in finite samples the selected class means invert below it, so that a nearest-mean classifier converges to the sign-flipped Bayes rule. The same population threshold holds for ridge-logistic regression. For a misaligned score we derive the selected means and the exact clean risk, so the sign flip becomes a continuous degradation with a computable above-chance boundary, and closed-form extensions cover independent score noise and a uniform-first random quota. On CIFAR-10, loss-ranked selectors do not invert real class geometry, but they displace class means four to six times farther than random selection does, consistently toward the nearest competing class. Two interventions on EL2N at a 5% fraction recover between a third and two thirds of its lost accuracy: scoring after 20 probe epochs instead of 1 raises final accuracy from 14% to 32%, and, in a separate paired experiment, replacing half of the hard quota with random draws under matched initialization raises it from 15% to 45%, against 62% for random selection. Only the second intervention moves the fixed-feature margin in step with accuracy. A geometry diagnostic that explains one successful intervention need not explain another, so such diagnostics should be validated across interventions before they are used to explain selection quality.

Primary area:
learning theory
Keywords:
data selection, coreset selection, data pruning, hard-example mining, Gaussian mixture, phase transition, random sampling, EL2N

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SfjfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims to discuss the method of hard-example selection, where simply based on highest loss if not enough, and the authors provide some studies over Gaussian distribution and real dataset. The goal is to better understand why the random/uniform selection still be the competitive baseline or even outperform those data selection methods.

Critical strengths

None

Critical weaknesses
  1. The clarity of the paper is unclear, e.g., what is the motivation of data selection, and how it affects the possible downstream applications, e.g. data selection during active learning.

  2. The presentation needs significant improvement, e.g. the paper presentation is lack of clarity, causing hard to follow. In abstract, no mentioning what is the purpose to select data at all; or in the introduction, what is the p* function? and what is phi function at line 49?

  3. Following point 2: In the section starting at line 154, it still does not clearly state what is the setup, it only mentioned what data is used but not used for what. many symbols are unclear, E.g., the defined rho is not referred but I guess it means 0.05 and 0.1 at line 158.

Questions for authors

The presentation flow needs to be significantly improved to better convey the paper's findings and novelty.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer Sfjf

Thank you for the review. Your comments center on the presentation, and the presentation has been rewritten following each of them. We answer in the order of your review and state what changed.

Motivation and downstream use. The abstract and the introduction now start from why data is selected. Training on a small subset of the training set saves computation. A common rule keeps the examples that a briefly trained network finds hardest, on the view that they are the most informative. When only a small fraction is kept this rule fails: with 5% of CIFAR-10 kept, a ResNet-18 trained on the hardest images reaches 14.1% test accuracy against 62.0% for a random subset. The paper asks why. The abstract no longer contains symbols. Active learning is a different problem, since it chooses which unlabeled examples to label, whereas we select from a labeled training set. Related work now says so in one sentence, so that readers do not expect results on it.

Undefined symbols in the introduction. The introduction now states the threshold in words, and every symbol in the main text is defined where it first appears. Φ is the standard normal distribution function. p*(r) is the threshold retained fraction below which, in the two-class Gaussian model, the two selected class means exchange positions. It is set only by how far apart the two classes are, measured by r = μ/σ. Formally p*(r) = Φ(z*), where z* is the unique solution of λ(z) = r and λ(z) = φ(z)/Φ(z) is the inverse Mills ratio, with φ the standard normal density.

Setup section (line 154 onward). Setup now comes directly before the experiments and says what each ingredient is used for, in three headed paragraphs. Subsets: a scoring probe scores every training image, the EL2N subset keeps the highest-scoring images of each class, and the random subset is drawn uniformly. Training: a ResNet-18 is trained from scratch on each subset and judged by its final test accuracy. Class means: the two quantities measured later, the count of selected class means that cross the midpoint and nearest-mean accuracy, are defined here before they are used. Your reading of the fraction symbol is correct: it is the retained fraction, with values 0.05 and 0.1. It is now written f throughout the main text, and Setup states how it relates to the p of the Gaussian model in Section 3 (the two are equal for the EL2N subset). The Experiments section opens with a short paragraph that says what each subsection does.

What the paper finds and what is new. In each class, almost all images kept by the hardest-example rule are ones that the scoring probe assigns to other classes (about 96%, against about 49% of a random subset), so the mean of their features lies toward those classes. Of the 30 selected class means (10 classes, 3 seeds), 22 are closer to the nearest competing class than to their own class, and none are for the random subset. Three ways of changing which images are kept all raise accuracy, and the position of the class means orders the repaired subsets almost as their test accuracy does. The same failure appears on four datasets and four architectures. The failure itself had been reported. Paul et al. saw accuracy drop sharply at extreme pruning rates, and Zheng et al. found that methods keeping the highest-scoring examples fall below random selection at high pruning rates, which they attributed to worse coverage of the data. What is new is that the paper traces the failure to the position of the class means of the kept images, measures that position directly, and gives a two-class Gaussian model in which the threshold fraction can be computed.

Revision. In summary, the abstract and the introduction open from the motivation of saving computation, the abstract has no symbols, related work has the sentence on active learning, each symbol is defined where it first appears, Setup is split into Subsets, Training and Class means with a single symbol f for the retained fraction, and Experiments opens with a roadmap. The presentation has changed substantially, and we would be grateful if you would read the revised PDF and reconsider your rating.

AuthorsOpenReview ↗

Follow-up to Reviewer Sfjf

Dear Reviewer Sfjf,

The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.

In brief: the presentation has been rewritten following each of your comments. The abstract and the introduction now open from why data is selected, the abstract has no symbols, every symbol is defined where it first appears, and Setup states what each ingredient is used for.

If anything remains unclear or unresolved, please let us know and we will respond before the deadline.

Thank you again for your time.

AuthorsOpenReview ↗

Reminder to Reviewer Sfjf

Dear Reviewer Sfjf,

A brief reminder that the discussion period closes on Nov 18 (AoE), about a day from now.

The revised abstract and introduction are the quickest way to see the changes made in response to your comments. If they read clearly to you now, we would be grateful if you would reconsider your rating. We can still answer any remaining question before the deadline.

Your comments on presentation made the abstract and the introduction easier to read. We know this is a busy period, and we are grateful for your time.

Reviewer_GcBJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when selecting the highest-loss examples fails at small retained fractions. In a balanced Gaussian model with signal-to-noise ratio r, top-loss selection against a fixed aligned reference inverts the selected class means below p*(r) related to the inverse Mills ratio; Theorem 1 establishes this with high probability in finite samples (controlling the empirical cutoff and class counts) and shows the nearest-mean risk converges to the sign-flipped Bayes risk, while a random subset is well behaved. Theorem 2 gives exact clean risk under a misaligned scoring direction, with an above-chance boundary p*(r/c). The empirical part measures selected-class-mean displacements in a fixed CIFAR-10 probe space, finds no sign inversion but 4–6× larger boundary-ward displacement for loss-ranked selectors, and studies two interventions on EL2N at 5%: later scoring (14%→32%) and a half-random quota (15%→45% vs 62% for random). Only the second moves the fixed-feature margin in step with accuracy, so the authors conclude that geometry diagnostics must be validated across interventions.

Critical strengths

(1) The theoretical results are correct, self-contained, and numerically verified: the threshold equation, the four simulated flip points (agreement to 5e-4), and the closed-form risk under mismatch all check out. (2) The writing is unusually candid: the scope of every claim is stated, negative results are reported as such, and the distinction between a paired intervention and a protocol-sensitivity study is made explicit. (3) The empirical protocol is careful (3 seeds, sample std, index files, matched recipes, same-batch controls) and the "diagnostic must be validated across interventions" message is a useful methodological contribution.

Critical weaknesses

(1) The core prediction (inversion) is unobservable in the chosen probe space by construction, since the probe's linear head classifies all training points correctly; the paper acknowledges this but still lists "do not invert" as a finding, so Theorem 1 is not actually tested. (2) The only quantitative prediction that is tested — displacement scaling with β_p — fails at epoch 20 (R² = −0.030) and is only moderate at epoch 1 (0.830). (3) The paper does not explain its motivating phenomenon: EL2N at 5% is 14% vs 62% for random, while the margin retention is 0.930; the diagnostic cannot account for a 48-point gap. (4) Scope is narrow: one selector, one dataset, two fractions for the interventions; one-dimensional Gaussian theory with nearest-mean/ridge-logistic learners only. (5) Code/index files are only promised "upon publication", so the artifact-based claims cannot be checked during review.

Questions for authors

The Gaussian analysis is correct and carefully verified, but it answers a narrower question than the title promises: the predicted sign inversion cannot be observed in a probe space whose linear head classifies every training point correctly, the only tested quantitative prediction (displacement ∝ β_p) fails at scoring epoch 20 (R² = −0.03), and neither the 48-point gap between EL2N and random at 5% nor the accuracy/margin decoupling is explained. The empirical scope is one selector, one dataset and two fractions, and the code and index files are promised only upon publication. OrderDP — a pool-plus-top-q selector that loses only 1.5 points at 30% selection — and InfoBatch are never discussed, although both are exactly the comparisons that would situate the failure. Measuring inversion in a subset-trained probe, explaining the 48-point gap, testing OrderDP at 5–10%, and releasing the artifacts are needed before the general claim can stand. I’ll consider raising the score depending on how the concern can be addressed.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer GcBJ (1/2)

Thank you. Your point about where inversion can be observed led to what is now the main result of Section 5.1, and your point (3) and the OrderDP comparison led to two of the three repairs in Section 5.2.

(1) Inversion in the probe space. Measured where it can appear, in the features of a scoring probe and in the subset-trained network you proposed, the crossing is seen directly. The submission measured class means in the penultimate features of a network trained for 40 epochs on the full training set. That network classifies every training image correctly, so, as you note, no crossing of the midpoint can be seen there (0 of 30 selected class means). Following your suggestion, we measured in two other feature spaces.

Space one: the features of the scoring probe (a further network trained with the same setting as the ten scoring probes, seed 0). At f = 0.05, 22 of the 30 selected class means of the EL2N subset cross the midpoint, and 20 at f = 0.1 (random subset: none). Nearest-mean accuracy is 13.8% with the selected class means of the EL2N subset, against 48.9% with those of the random subset and 49.2% with the full-data class means. The crossing appears in all three subset seeds, and in the features of two further scoring probes trained with other seeds (18 of 30 at both fractions, random subset: none). That images the scoring probes misclassify lean toward other classes in such a probe is expected. Beyond that, whether a class mean crosses depends on the fraction kept among the ranked images (pool-then-top keeps the highest scores of a random pool, and 6 of 30 cross), and the crossing remains in a network that took no part in scoring (space two) and on three other architectures (general response).

Space two, the one you proposed: the features of the network trained on the EL2N subset, with midpoints between full-data class means in that network's own features. Here 10 of 30 selected class means cross and nearest-mean accuracy is 14.7% (random subset: 0 of 30 and 60.4%). Choosing between automobile and truck on their test images, this network is right 43% of the time, and between deer and horse 41%, below chance (50%) in all three seeds. Of the 45 class pairs, 11 are below one half in every seed (15 in the three-seed mean). Networks trained on a random subset with equal class sizes reach 86% and 81% on these two pairs. Revision: these measurements are the main result of Section 5.1, and "do not invert" is removed.

(2) The β_p scaling test. We agree with your reading: moderate support at epoch 1 and a failure at epoch 20. The prediction assumes that the score is linear along one direction of the fixed feature space. The loss of a 20-epoch probe is far from linear in those features, and the fit fails there (R² = −0.030, against 0.830 at epoch 1). The result marks where the linear-score model stops applying. The revision builds Section 5.1 on the direct measurements under (1) and moves the scaling test to the appendix ("Measurements in the features of the full-data network").

Accuracy/margin decoupling. The submission's result stands and is kept in that appendix. It is a property of the full-data features, which do not separate these subsets. In the features of the scoring probe, scoring later improves class-mean position too: at f = 0.05 and scoring epochs 1, 10 and 20, crossings are 22, 17 and 13 of 30, nearest-mean accuracy 13.8%, 34.9% and 40.3%, and test accuracy 15.0%, 25.1% and 32.1%. (15.0% and the general response's 14.1% are the same subsets in two independent sets of runs.) Section 5.3 tests this on all three repairs, the cross-intervention check the submission called for.

(3) The 48-point gap. The margin retention of 0.930 was measured in features that cannot see the structure of the EL2N subset. In the features of the scoring probe that structure is visible and consistent with the size of the gap. Every one of the 2,500 images in the EL2N subset at f = 0.05 has an EL2N score above √0.9, the largest score that an image classified correctly by the scoring probes can have with ten classes. The scoring probe whose features we use misclassifies about 96% of the EL2N subset, against about 49% of the random subset. The EL2N subset of a class therefore consists almost entirely of images that the probe assigns to other classes, and their mean lies toward those classes (22 of 30 crossings). Like the network trained on the EL2N subset, these class means classify barely better than chance.

The random share points the same way. Filling a share of each class with random images in place of the highest-scoring ones, from the same initial weights, gives 15.0, 28.9, 45.2, 52.5 and 61.7% at f = 0.05 for shares 0, .25, .5, .75 and 1, and 32.4, 52.0, 58.4, 71.5 and 72.7% at f = 0.1. Over shares 0, .5 and 1 at f = 0.05, the share of images with EL2N score above √0.9 falls from 1.00 to 0.62 to 0.25.

AuthorsOpenReview ↗

Reply to Reviewer GcBJ (2/2)

OrderDP at 5 to 10%. Its pool-then-top rule, applied once, recovers most of the lost accuracy, the Gaussian model says why, and it still stays below random. OrderDP draws a random pool and keeps its highest-loss part. What matters in Theorem 1 is the fraction p kept among the points that are ranked. For the EL2N subset p equals the retained fraction f. For a rule that keeps the top 60% of a random pool, OrderDP's default setting, p is 0.6 whatever f is. Inversion requires p to be below the threshold p*(r). At r = 1 the threshold is about 0.38, so p = 0.6 is above it, and the threshold exceeds 0.6 only when the two classes overlap heavily (r below about 0.64). We tested this with pool-then-top: in each class, draw a random pool of size f/0.6 and keep the 60% with the highest EL2N scores. Final test accuracy is 53.3 ± 5.6% at f = 0.05 and 69.6 ± 0.6% at f = 0.1, against 15.0 and 32.4% for the EL2N subset and 61.7 and 72.7% for the random subset. The share of images with EL2N score above √0.9, all of which at least one scoring probe misclassifies (part 1, point 3), falls from 1.00 in the EL2N subset to 0.41 (random subset: 0.25). OrderDP itself reselects repeatedly during training and we select once, so the experiment tests its pool rule. Section 5.2 of the revision reports it as the second repair, and related work discusses OrderDP.

InfoBatch. InfoBatch randomly prunes a share of the low-loss examples, rescales the gradients of the remaining ones, and anneals back to the full data late in training. It never trains only on the hardest examples, which is the regime our theory and experiments address. Related work now discusses it alongside OrderDP.

(4) Scope. The failure and the crossing replicate on other datasets and architectures. Each dataset uses ResNet-18, its own scoring probes, 200 epochs and final test accuracy. Class means are measured in the features of the scoring probe.

f = 0.05, EL2N subset vs random subsetCIFAR-10CIFAR-100Tiny ImageNetImageNet-100
Test accuracy14.1 ± 1.8 vs 62.0 ± 1.16.0 ± 0.6 vs 23.3 ± 0.35.3 ± 0.1 vs 21.7 ± 0.611.0 ± 0.5 vs 38.5 ± 0.9
Selected class means crossing the midpoint22/30 vs 0/30120/300 vs 9/300196/600 vs 11/600133/300 vs 0/300
Selected class means still closest to their own class7% vs 100%12% vs 94%10% vs 96%27% vs 100%
Nearest-mean acc., EL2N subset / full-data class means13.8 / 49.26.5 / 19.34.5 / 13.912.3 / 28.5

On every dataset most selected class means of the EL2N subset are no longer closest to the full-data class mean of their own class (third row). With 100 or 200 classes a selected class mean that moves away from its own class need not head for the nearest competing class, so the share that crosses the midpoint is lower on the three new datasets than on CIFAR-10, from about a third to almost a half. Nearest-mean accuracy falls to less than half of its full-data value on every dataset. At f = 0.1, ImageNet-100 gives 26.0 ± 1.2 against 53.7.

Architectures: with ResNet-50, MobileNetV3-Large and InceptionV3 on CIFAR-10, each scored by probes of its own architecture, the failure and the crossing appear at f = 0.05 (general response). At f = 0.1 the EL2N subset stays far below the random subset (21.4 against 70.2, 22.8 against 67.0 and 25.8 against 72.0).

The other items of your point (4):

  • Selectors. Scoring later also raises accuracy with GraNd (27.4% to 34.6% at f = 0.05). Over eleven selection rules, measured in the features of the scoring probe (Table 10), the rank correlation between crossing count and test accuracy is −0.92 at f = 0.05 and −0.94 at f = 0.1, and every rule with a crossing ranks below every rule without one.
  • Datasets. Scoring later also raises accuracy for EL2N on CIFAR-100 (6.0%, 7.8% and 11.0% at scoring epochs 1, 10 and 20).
  • Fractions. At f = 0.05, 0.1, 0.3, 0.5 and 0.7, the EL2N subsets of Figure 2 (right), scored separately at each f, have 20, 19, 15, 6 and 5 of 30 crossings (random: none) and are 47.3, 40.0, 10.1 and 0.5 points below the random subsets at the first four (Table 12).
  • Theory. The model is a two-class Gaussian model, and the revision uses it as a reference for the experiments: Section 3 says which event to look for and Section 5 looks for it on real images. Theorem 2 is stated in d dimensions, and the population threshold of Theorem 1 holds for isotropic Gaussian mixtures scored along their signal direction (Remark 1, appendix).

(5) Code and indices. Code, selected indices, scores and per-run logs for every experiment in the paper and in this response are at https://anonymous.4open.science/r/hard-example-selection-diagnostics/

You wrote that you would consider raising the score depending on how the concerns are addressed, and we hope the measurements above and the revised paper allow you to do so.

AuthorsOpenReview ↗

Follow-up to Reviewer GcBJ

Dear Reviewer GcBJ,

The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.

In brief: the inversion is now measured where it can appear. 22 of 30 selected class means cross the midpoint in the features of a scoring probe, and 10 of 30 in the subset-trained network you proposed, against none for a random subset. OrderDP's pool-then-top rule is tested at 5% and 10%, and code and selected indices are released at the anonymous link.

If anything remains unclear or unresolved, please let us know and we will respond before the deadline.

Thank you again for your time.

Reviewer_hm4aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes the asymptotic error under score-based pruning when considering nearest-mean classification in a binary GMM setting with NN training points. In other words, keeping only a fraction p∈(0,1)p \in (0, 1) of the training points with the highest loss. It is presented under a perfectly aligned scorer in 1D (where a threshold p∗p^{\ast}, determined by the class separation rr, dictates whether the classifier is close to the worst possible error or to the best possible error, while random selection is always close to the best possible error), and the general case of an imperfect scorer, where the distance from the best possible error is quantified. Experiments on CIFAR-10 and ResNet-18 architecture are performed. Three different scoring methods are used (e.g., EL2N, which, using a probe network, quantifies the score as the distance between the predicted probabilities and the true label). Only the highest-scoring images per class (5% and 10%) are kept, and then a fresh ResNet-18 is trained on them. Random selection gives much higher accuracy, which is consistent with the theory. Class averages shift toward other classes much more than in random selection, and, under the EL2N scoring method, two accuracy improvements are considered: using more probe epochs, and half-random: replacing half of the chosen examples with random ones. Only half-random lowered the averages' shift, which, as the authors argue, shows that the shift alone cannot explain the accuracy failure.

Critical strengths
  1. A precise theoretical analysis. Theorems 1-2 and Lemma 2 explicitly analyze the asymptotic risk caused by hard-example selection, for a perfectly aligned scorer, an imperfect scorer and a noisy scorer with partial random selection. The numerical simulations are aligned with the theory. Together, these make the theoretical analysis reliable.

  2. The experiments support some theoretical predictions. Multiple scoring methods (EL2N, GraNd, Forgetting) on the CIFAR-10 dataset (at a 5% fraction) shifted class averages toward a different class much more than random selection. This shows that some properties of the toy model also appear in real data.

Critical weaknesses
  1. Theorem 1 doesn't add much beyond known theory. As the paper acknowledges, it is well known that the error grows when using a small fraction of the training set, starting from the hardest examples. Theorem 1 shows that the decision rule flips, and characterizes the flip carefully for finite data as well as analyzes when this happens, which follows from a standard calculation. Thus, Theorem 1 has limited novelty.

  2. The main empirical conclusion is drawn from a narrow setting. The authors conclude that the accuracy drop cannot be fully explained by class averages shifting toward neighboring classes. It is only drawn from experiments done on the CIFAR-10 dataset, EL2N method, ResNet-18 architecture and two fractions of kept data (0.05 and 0.1). The counterexample, which the conclusion rests on, is only done on the EL2N method. Repeating the experiments with different datasets, architectures, methods, and expanding the fractions range will better reflect whether this finding generalizes.

Questions for authors
  1. Clarification. Dohmatob, Pezeshki & Askari-Hemmat ("Why Less is More (Sometimes): A Theory of Data Curation," ICLR 2026; arXiv:2511.03492) analyze a similar setup (error formulas under score-based pruning). How do your results differ? Please cite this paper and discuss it.
  2. Additional experiments. Can you perform the scoring-later comparison with additional datasets other than CIFAR-10 and additional methods, like GraNd or Forgetting, and report back in the paper whether the mean shift fails to track accuracy?
  3. Limited analysis. Is there a different statistic of the selected training data that tracks accuracy under both fixes (later scoring and random quota)?
  4. Clarification. Currently, each subset is trained for a fixed number of 200 epochs. Would training each subset for more epochs (the same number steps as training on the full CIFAR-10 dataset) change the gap between random selection and the EL2N method?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer hm4a

Thank you for the pointer to Dohmatob et al. and for your four questions. Each led to something new in the revision: the comparison with their model, scoring later with GraNd and on CIFAR-100 (Q2), the ordering of Section 5.3 (Q3) and equal-step training (Q4).

Theorem 1 and what is new. We agree that the threshold p*(r) follows from a standard truncated-Gaussian calculation. The revised introduction says so and uses the model to tell the experiments which event to look for, and Section 5.1 now observes that event. The part we do consider non-standard is Theorem 2: once the scoring direction is misaligned the classifier matters, and nearest-mean and ridge-logistic classifiers fitted to the same selected population can land on opposite sides of chance (risk 0.459 against 0.592 in the appendix example).

Relation to Dohmatob, Pezeshki and Askari-Hemmat (ICLR 2026, as the reviewer notes). They give test-error formulas for a ridge learner in the proportional limit d/n → φ, and we give an explicit threshold in the retained fraction and a finite-sample statement. In their model the training label is a deterministic sign of the input, y = sign(x^T w_g). Their Theorem 2 shows that keeping hard examples is optimal when generator and pruner are both good, while keeping easy ones wins when the generator is poor and the pruner is good. Because every positive training point satisfies x^T w_g > 0, the positive-class mean of any selected subset stays on the correct side of the boundary that generates the training labels. In our two-class Gaussian model the classes overlap, keeping the highest scores keeps precisely the points on the wrong side of the Bayes boundary, and below p*(r) these points dominate each selected class and pull its mean across the midpoint.

Q2, scoring later with other methods and datasets. Scoring later also helps GraNd and CIFAR-100. Nearest-mean accuracy in the scoring probe's features rises with test accuracy, and the submission's margin retention (40-epoch network) does not. All at f = 0.05 (Table 11, Sections 5.2 and 5.4):

Scoring epoch11020
GraNd, CIFAR-10: test accuracy27.4 ± 1.328.1 ± 0.734.6 ± 0.9
Nearest-mean accuracy29.337.041.5
Margin retention0.9410.8970.909
EL2N, CIFAR-100: test accuracy6.0 ± 0.67.8 ± 0.311.0 ± 0.8
Nearest-mean accuracy6.59.010.9

The random subset on CIFAR-100 reaches 23.3 ± 0.3. Over the GraNd subsets and Q3's eight, nearest-mean and test accuracy have rank correlation 0.91. Forgetting fits the same pattern: its subset has 11 of 30 crossings and 33.49% test accuracy, and over eleven selection rules, the more class means cross, the lower the test accuracy (rank correlation −0.92, −0.94 at f = 0.1, Table 10).

Q3, a statistic that tracks accuracy under both fixes. Yes: the position of the selected class means, measured in the features of the scoring probe. We compare eight subsets (f = 0.05): the EL2N subset, four subsets in which a share 0.25, 0.5, 0.75 or 1 of each class is filled with random images (share 1 is a random subset), pool-then-top (the highest-scoring 60% of a random pool), and EL2N scored at epochs 10 and 20. Over the eight, nearest-mean accuracy and test accuracy have Spearman rank correlation 0.95 (0.86 at f = 0.1). The EL2N subset, the least accurate, has 22 of 30 selected class means crossing the midpoint, and the random subset, the most accurate, has none. In the submission's 40-epoch network none of the eight has a crossing and nearest-mean accuracy is 85.7% or 85.8%, so it could not separate them. Within each repair, class-mean position and test accuracy move together, and across repairs the orderings largely agree (Section 5.3).

Q4, matching the step count of full-data training. No: the failure is in the subset, not in training length. Training both subsets for 4,000 epochs gives 40,000 steps, against 39,200 for 200 full-data epochs. The EL2N subset moves from 14.1% to 20.1% and the random subset from 62.0% to 67.5%, so the gap goes from 47.8 to 47.4 points.

Weakness 2 and revision. Beyond Q2 and Q4, Section 5.4 adds datasets and architectures: on CIFAR-100, Tiny ImageNet and ImageNet-100, and with ResNet-50, MobileNetV3-Large and InceptionV3 on CIFAR-10, the EL2N subset reaches less than a third of the random subset's test accuracy at f = 0.05 (general response). At f = 0.05, 0.1, 0.3, 0.5 and 0.7, the EL2N subsets of Figure 2 (right), scored separately at each f, have 20, 19, 15, 6 and 5 of 30 crossings (random: none) and are 47.3, 40.0, 10.1 and 0.5 points below the random subsets at the first four (Table 12).

Reviewer_hm4aOpenReview ↗

Response to authors’ rebuttal

I thank the authors for the thorough answers and revision. The new experiments address my questions: the comparison with Dohmatob et al. is clear, the equal-step runs show that the gap is not caused by training length, and the replication on further datasets, architectures and GraNd addresses my concern about the narrow setting. I keep my score at 3, weak accept.

AuthorsOpenReview ↗

Reply to Reviewer hm4a

Dear Reviewer hm4a,

Thank you for reading the revision and for confirming that the new experiments answer your questions. Your questions on training length, on GraNd and on other datasets led to the equal-step runs and to Section 5.4, and the comparison with Dohmatob et al. is clearer because you asked for it. We are grateful for your time and for your support of the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers. GcBJ's point about where inversion can be observed led to the main result of Section 5.1, and hm4a's questions led to the equal-step, GraNd and CIFAR-100 runs and the ordering of Section 5.3. In short: (1) The predicted event is directly observable. (2) The failure is not specific to CIFAR-10 or ResNet-18: it replicates on three more datasets and three more architectures. (3) It lies in the subset, not in training length. (4) Scoring later also raises accuracy for GraNd and on CIFAR-100. Unless stated, runs use 3 seeds and CIFAR-10. Accuracy is final test accuracy, mean ± sample SD.

Setup. A scoring probe is a ResNet-18 trained for one epoch on the full training set, and the EL2N score of an image is the distance between its output probabilities and the one-hot label, averaged over ten probes. The EL2N subset keeps the highest-scoring images of each class at a retained fraction f of 0.05 or 0.1, and a ResNet-18 is trained on it from scratch. In a two-class Gaussian model, Theorem 1 gives a threshold fraction below which the two selected class means exchange positions (inversion). With many classes the corresponding event is that a selected class mean crosses the midpoint, meaning it is closer to the full-data class mean of the nearest competing class (the class whose full-data mean is nearest its own) than to that of its own class.

Selected class means cross the midpoint in the features of the scoring probe. The submission measured class means in the features of a network trained for 40 epochs on the full training set. That network classifies every training image correctly, and no crossing is visible there. The limitation was where we measured. Prompted by GcBJ, we measured in the features of a scoring probe, and there the crossing appears directly. Of the 30 selected class means of the EL2N subset (3 seeds, 10 classes), 22 cross the midpoint at f = 0.05 and 20 at f = 0.1, against none for the random subset. Nearest-mean accuracy, which classifies test images from class means alone, is 13.8% with the selected class means of the EL2N subset, against 48.9% for the random subset and 49.2% with the full-data class means. Nor is this confined to the networks that gave the scores: in the features of the network trained on the EL2N subset, 10 of 30 selected class means still cross (random subset: none). Choosing between automobile and truck on their test images, this network is right 43% of the time, and between deer and horse 41%, below chance (50%) in all three seeds.

The failure replicates on other datasets and architectures. ResNet-18, f = 0.05, each dataset with its own scoring probes; counts pool 3 seeds:

DatasetEL2N subsetRandom subsetSelected class means crossing (EL2N, random)
CIFAR-1014.1 ± 1.862.0 ± 1.122/30, 0/30
CIFAR-1006.0 ± 0.623.3 ± 0.3120/300, 9/300
Tiny ImageNet5.3 ± 0.121.7 ± 0.6196/600, 11/600
ImageNet-10011.0 ± 0.538.5 ± 0.9133/300, 0/300

On every dataset at most 27% of the selected class means of the EL2N subset are still closest to the full-data class mean of their own class, against at least 94% for the random subset. On CIFAR-10 with scoring probes of the same architecture, ResNet-50 gives 9.4 ± 1.1 against 53.5 ± 2.1 with 12 of 30 crossing, MobileNetV3-Large 13.8 ± 0.3 against 55.6 ± 1.1 with 19 of 30, and InceptionV3 9.6 ± 0.4 against 50.0 ± 1.7 with 23 of 30 (random subsets: none).

The failure is in the subset, not in training length (hm4a Q4). Training both subsets for 4,000 epochs, about as many steps as 200 full-data epochs, raises the EL2N subset from 14.1% to 20.1% and the random subset from 62.0% to 67.5%. The gap moves from 47.8 to 47.4 points.

Scoring later also helps GraNd and CIFAR-100 (hm4a Q2). Scoring at epoch 20 instead of 1 raises GraNd at f = 0.05 from 27.4% to 34.6%, and EL2N on CIFAR-100 from 6.0% to 11.0%.

Code, indices, scores and per-run logs: https://anonymous.4open.science/r/hard-example-selection-diagnostics/

Changes in the uploaded revision. (i) Section 5.1: midpoint crossing measured in the features of the scoring probe and of the network trained on the subset, in place of the "do not invert" statement, and equal-step training. (ii) Section 5.2: three repairs (GcBJ, hm4a), the random share at five values, pool-then-top, and scoring later including GraNd. (iii) Section 5.3: class-mean position ranks eight subsets almost as test accuracy does (rank correlation 0.95, hm4a Q3). (iv) Section 5.4: three more datasets, three more architectures, scoring later on CIFAR-100. (v) Related work: Dohmatob et al., OrderDP, InfoBatch and the difference from active learning. (vi) Following Sfjf, the abstract and introduction start from why data is selected, the abstract has no symbols, every symbol is defined at first use, and Setup is rewritten.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.