- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies coreset selection and asks how the data budget and the selection score interact, and what role each plays. It compares two selectors. LFrac keeps the easy samples that a small proxy network learns early and stably, and Herding keeps a representative spread of samples. LFrac wins at small budgets and Herding at larger ones, and the budget where they swap places is called m₀. The authors argue that m_0 is better described in absolute samples per class than as a fraction of the dataset. With two controlled interventions, shrinking the per-class pool and degrading image resolution, and with negative results showing that m_0 cannot be predicted from dataset descriptors, that blending the selectors does not help, and that CCS helps only near m_0.
Critical strengths
(i)The paper is well written and motivated (ii)Reporting budgets as fractions is universal. Asking which coordinate is stable across datasets is a worthwhile question for the field. (iii)The pool-shrinking experiment is the strongest contribution and demonstrably interesting in Sec experiments. (iv)Highliting that simple descriptors, naive mixing, and an oracle-tuned CCS do not substitute for measuring the boundary is informative
Critical weaknesses
(i)The paper makes three kinds of claims : a phenomenon (the crossover exists), a coordinate (samples per class), and a set of negative results (no predictor, no mixing gain). It is not obvious if these three are the main contribution.
(ii)Intervention in image space" remains unclear as a motivation. Coreset method operates on a set of samples, yet the paper frames the visual-bandwidth experiment as an intervention on the image. The why is not motivated clearly. The experiment downsamples and upsamples back, so both selection and training see degraded inputs. The paper concedes that pixel count is not a calibrated measure of classification-relevant information and that the result does not falsify hypothesis (ii), that the boundary tracks per-image information. The experiment therefore cannot separate the hypotheses it was meant to separate. Its role in the argument, and why it counts as an intervention rather than a robustness check, is never made clear.
(iii)All conclusions concern LFrac vs. Herding with ResNet-18. The paper acknowledges that the numerical value of m_0 depends on the selector pair and feature source, and that the evidence does not identify its cause. Yet the title and the Sec 5 recommendations are stated in general terms. The abstract's "winning selection principle changes with budget" is a statement about principles, but two implementations are tested.
Questions for authors
(i)Which of the three contributions (phenomenon, coordinate, negative results) is the primary one? If forced to drop one, which would it be? (ii)Why is the resolution experiment described as an intervention in image space, and which of hypotheses (i)-(iii) does it actually distinguish? (iii)Does m_0 remain in the same samples-per-class band if LFrac is replaced by a confidence-, forgetting-, or EL2N-based easy-sample selector, or Herding by another coverage method?
3 comments
Response to follow-up questions (1/2)
Thank you for the follow-up. We answer the six points in order. The per-pool EL2N numbers in point 2 come from a small set of new runs (3 seeds, ResNet-18, protocol as in the paper).
1. Within-dataset vs. cross-dataset: the argument needs only a dataset-specific m0. The claim is that for a given dataset, selector pair and learner, the boundary is set in samples per class and fractions fail to track it. It does not require m0 to be shared across datasets, and the values you list (about 46 on ImageNet-1k, 65 to 80 on ImageNet-100) already differ. The 5% example uses each dataset's own boundary. The same 5% is 25 per class on CIFAR-100, on the easy-first side of its boundary, and about 64 per class on ImageNet-1k, past its boundary of about 46. Comparing methods at a matched fraction across datasets can compare them in different regimes. Inside one dataset the pool-shrinking experiment shows the same effect: on ImageNet-100 the boundary sits at about 5% of the full pool and at about half of the smallest pool. On whether samples per class is the right unit: the pool-shrinking design rules out fractions, and samples per class is the reading that stays in a band when the pool shrinks. A boundary stated per class carries over when the pool of the same dataset changes size, and one stated as a fraction does not. The paper claims no more than that.
2. Pool-independent instrument for EL2N: under the per-pool protocol the band is wider, and m0 still does not shrink in proportion to the pool. First a clarification. In the fixed rows of our first-round table, LFrac and EL2N-easy both use a full-pool proxy, so that comparison was under one protocol. A full-pool proxy gives LFrac 3 to 5 points higher accuracy on the small pools than a per-pool proxy, which strengthens the easy side and moves m0 up, away from the fraction prediction. The per-pool protocol is the stricter test, and we ran it. EL2N-easy now uses one 10-epoch EL2N proxy per pool and keeps the lowest m scores per class. Paired with Herding on CIFAR-100, N = 500 / 250 / 150 / 100 per class:
| Herding extractor | m0 (EL2N-easy, per-pool proxy) | Fraction prediction | Slope |
|---|---|---|---|
| per-pool (paper protocol) | 45 / 69 / >70 / 39 | 45 / 22 / 13 / 9 | +0.12 (three uncensored pools) |
| fixed full-pool | 61 / 61 / 54 / 47 | 61 / 31 / 18 / 12 | +0.16 |
Two curves cross more than once in the transition region, N = 500 in the first row (first 45, last 59) and N = 150 in the second (first 54, last 68). The others cross once or are censored. The per-pool band is wider than under the fixed proxy for the reason given to Reviewer ghPW in our first response: per-pool Herding redraws its extractor on each call and sits about one point from its competitor in the transition region, so its crossing is read with noise. With a fixed extractor all four pools fall between 47 and 61. Under both pairings the smallest pool lands at about 4 times the fraction prediction. The position and width of the band depend on the instrument, and no instrument makes m0 shrink in proportion to the pool. For comparison, per-pool LFrac paired with fixed Herding gives 59 / 50 / 37 / 49 on the same pools, a similar band from a different easy-first score.
3. Class-balanced random in place of Herding: the slope of 0.08 uses the three uncensored pools and assigns no value to the fourth. On those pools (N = 1267, 600, 300) m0 is 125, 105 and 111. On the N = 150 pool the grid ends at 90 per class and LFrac still leads class-balanced random by 0.01 points there, so the crossing is at or just above 90. Assigning 90 to that pool, the value most favourable to the fraction hypothesis, gives a slope of 0.13. Assigning 125, the largest value seen on the other pools, gives a slope of −0.01. The fraction hypothesis needs about 15 per class on that pool, and the difference is still positive at 90. Either treatment leaves the slope far from 1.
Response to follow-up questions (2/2)
4. Score definition: ties are broken by sort order, and they do not move the small-budget side. One correction: the shrinking-pool arms use K = 10, so the score takes 11 values (K = 5 is the window-sensitivity arm of Table 1 and Appendix A). Ties are broken by the order of the sorting routine, which is deterministic and reproducible but has no principle behind it, as Appendix C states. Ties are common at small budgets. At 25 per class on the full ImageNet-100 pool about 92% of the selected samples share the cut-point score, almost all of them correct in every epoch. At 20 per class on the full CIFAR-100 pool about two thirds do. The share falls as the pool shrinks, to about one third at 25 per class on the smallest ImageNet-100 pool, and LFrac leads Herding by 5 to 6 points at that budget on both pools. At small budgets LFrac selects a sort-ordered slice of the samples the proxy learned early. Three pieces of evidence say the slice does not matter. First, EL2N is continuous, with 49971 distinct values among the 50000 CIFAR-100 training samples, so it has almost no ties. Swapping it in for LFrac under the fixed instrument keeps the easy-first lead at small budgets, with m0 within 3 of LFrac on the two largest pools (numbers in point 5). Second, the selected members are interchangeable: LFrac selections from different proxy seeds overlap by only about 0.15 Jaccard, since the tied samples near the cut change from seed to seed, and downstream accuracy stays within seed noise (Appendix D). Which tied samples are taken does not change the result. The early-learned composition does. Third, the K = 40 arm of §4.1 has 41 score values and far fewer ties, and it gives the same crossing reading as K = 10, about 85 on the full CIFAR-100 pool with the Herding features of that section (Appendix A).
On forgetting: both scores have ties at the easy end, and the difference is the size of the tied block. For LFrac on ImageNet-100 the samples correct in every epoch are about a tenth of the median class, so a small budget taken from that block is still a draw from the easiest tenth. For forgetting computed from the same 10-epoch proxy, two thirds of the samples in the median class have zero forgetting events. A small budget taken from that block is close to a random draw from most of the class. Forgetting was excluded for lack of ordering power at those budgets.
5. Margin and the EL2N substitute: EL2N is margin-aware, and its m0 agrees with LFrac's on the two largest pools and comes earlier on the smallest, where EL2N-easy is the less accurate selector. EL2N is the L2 distance between the softmax vector and the one-hot label. It falls as the true-class probability rises and grows with the mass on wrong classes, so it carries margin information and measures a different quantity from LFrac's count of correct epochs. On the same dataset, pool and fixed instrument:
| N per class | 500 | 250 | 150 | 100 |
|---|---|---|---|---|
| m0, LFrac | 59 | 63 | >70 | >70 |
| m0, EL2N-easy | 61 | 66 | >70 | 60 |
| LFrac minus EL2N-easy accuracy, mean over budgets | −0.4 | +0.4 | +1.7 | +2.9 |
The claim does not require the two scores to give the same m0, because the value of the boundary depends on the selector pair. The table shows how. On the two largest pools the two selectors are within 0.4 points in accuracy and their crossings with Herding agree within 3. On N = 150 both are still ahead of Herding at the largest budget, so that pool cannot order the two crossings. On the smallest pool LFrac is 2 to 4 points more accurate than EL2N-easy across the budget grid. Both are compared with the same Herding runs, so LFrac stays ahead of Herding for longer and its crossing comes later. A stronger easy side moves the boundary up, as in our general response. Under either score the smallest pool crosses far above the 12 per class that a fixed fraction predicts.
6. Held-out prediction: the unit predicts how m0 responds to pool size, not its value. The prediction is that on a new dataset or selector pair, shrinking the pool does not pull m0 down in proportion, so the fraction reading drifts. The paper contains one out-of-sample test: ImageNet-100 was run first and CIFAR-100 was a later replication under the same protocol, with the predicted outcome. Each pairing added in the first round (fixed instruments, EL2N-easy, class-balanced random, VGG-16-BN) was a further test. In each a proportional drop was a possible outcome, and it did not occur. The value of m0 is not predictable at present. That is the negative result of Appendix E, and the paper says the boundary has to be measured per dataset.
The new per-pool EL2N numbers are now in the updated PDF (Appendix I, Table 8). The questions led to runs we had not done, and we will answer anything further during the remaining discussion period.
Response
I thank the authors for their response.
Within-dataset vs. cross-dataset The boundary is set in samples per class. The pool-shrinking design varies N within a dataset. It shows that m0 does not scale with the pool, but does it show that samples per class is the right unit ? The authors response own numbers (about 46 on ImageNet-1k, 65-80 on ImageNet-100, 60-66 for EL2N-easy on CIFAR-100) put m0 at different values per dataset and pair. You argue that 5% is 25 per class on CIFAR-100 and about 64 on ImageNet-1k, with the crossover at about 46. Does this argument require m0 to be shared across datasets, or only a dataset-specific m0 that fractions fail to track?
Pool-independent instrument for EL2N EL2N-easy uses a proxy trained once on the full pool and reused for every shrunken pool. LFrac trains its proxy per pool. Does the stable band hold under the per-pool protocol for EL2N too, or does fixing the proxy test an easier setting?
Pool-independent instrument for EL2N EL2N-easy uses a proxy trained once on the full pool and reused for every shrunken pool. LFrac trains its proxy per pool. Does the stable band hold under the per-pool protocol for EL2N too, or does fixing the proxy test an easier setting?
Random vs. Herding On the smallest ImageNet-100 pool, m0 lies above the largest tested budget. Is the slope of 0.08 computed with that point treated as censored, or is a value assigned to it? How would the slope change under each treatment?
Score definition For LFrac with K = 5, si takes only six values. How are ties broken in per-class top-k selection, and does this affect the small-budget side of the crossover? The rebuttal raises ties as a reason to exclude forgetting counts, but does not address them for LFrac.
** Margin and the EL2N substitute** The rebuttal treats confidence as a close relative of correctness rate. Since EL2N uses softmax distance to the one-hot label, is it a margin-aware score? If so, did m0 agree between LFrac and EL2N-easy on the same dataset and pool, so the two scores can be compared directly?
Held-out prediction and mechanism Can the stated unit predict m0 on a held-out dataset or selector pair?