- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper evaluates whether coreset selection methods are worth their cost. Previous works have treated the preprocessing cost as “free”. This paper tries to standardize downstream training and prices selection on a wall-clock axis that is estimated from timing probes. Results show that selection cost is dominated by a fixed full-dataset scan.
Critical strengths
- The paper is well motivated and easy to follow.
- Paper tries to provide practical insights, also it is appreciated to fix bugs as reported that was done in previous works. This is good for the community in general.
Critical weaknesses
-
Unclear contribution beyond prior work: RS2 already accounts for selection time and evaluates dataset distillation, noisy datasets, and GPT-2 retraining. The finding about hard examples at larger fractions also appears in prior work. The paper should clearly explain what it adds beyond these results.
-
Unclear practical takeaway and dependence on hardware: The paper does not clearly explain what its results tell users about which selection method to choose or what are the insights
Questions for authors
Clarity
Could you clarify the following statements?
“granting each selector its most favorable operating point under one recipe” — What is the most favorable operating point?
“We include RS2 as a first-class strategy” — What is a first-class strategy?
“Selection cost settles the cheapest anchors before accuracy enters” — What is an anchor?
“These compressed full-data points are the strategy equivalent to RS2 without replacement, listed as Full” — This is confusing. What do RS2 and Full refer to in the paper?
In Section 4.5, it is unclear what setting is being evaluated. Is the coreset being reused? If so, does this result in multiple instances of the same data point?
Soundness
The statement “Data Diet, DeepCore, and DRoP report accuracy at fixed fractions and treat selection as free,” Figure 1, and the contribution claiming that the paper introduces measuring preprocessing time are problematic: RS2 does not treat selection as free; this has already been done in that paper. Furthermore, RS2 also includes dataset distillation, noisy datasets (robustness analysis), and GPT-2 retraining. Therefore, it is unclear what the main contribution of the paper is.
What is the practical takeaway from the experiments in the paper? Is it that no single method wins in all settings? Or that all of these coreset selection methods incur unwanted costs?
Doesn’t the ratio of T_sel to T_train depend on the hardware used? Using a newer GPU could favor Random, as T_train can be reduced with newer hardware, such as Tensor Cores. However, coreset selection methods might not utilize the new hardware as fully as training does. Wouldn’t this make the ratio and the paper’s conclusions dependent on the selected hardware?
“At large fractions hard-example scoring often leads and the methods converge. Forgetting often ranks first at f≥0.5, which matches hard-to-learn examples becoming more useful at larger fractions.” — Isn’t this already shown in Sorscher et al.’s “Beyond Neural Scaling Laws”? Both theoretical claims and experimental evidence for this already exist. Does this result simply confirm that the finding holds in a different setting (model, dataset)?
6 comments
Reply to Reviewer ArUz (2/2)
This reply covers the practical takeaway, hardware, and the five terms. Reply 1/2 covered RS2 and Sorscher et al.
1. Practical takeaway
In the settings we evaluate, sophisticated one-shot selectors should not be the default: the accuracy they buy with selection time is smaller than what the same time buys when spent on training. Of your two readings, this is closer to "all of these coreset selection methods incur unwanted costs" than to "no single method wins in all settings", and it is more specific. At none of the budgets we compare does a sophisticated selector come out ahead.
The evidence is the comparison under 8 budgets each on CIFAR-10 and Tiny ImageNet (caps in seconds on selection plus training time, defined in item 3 below). Each method enters with the best cell (fraction f, E epochs) it can afford.
| Winner | Budgets won | Where |
|---|---|---|
| Full (all data, fewer epochs) | 8 | CIFAR-10: 5, Tiny ImageNet: 3 |
| RS2 | 6 | CIFAR-10: 913 s, Tiny ImageNet: 5 |
| Uniform (class-balanced random) | 2 | CIFAR-10: 64 s and 78 s |
| Any sophisticated selector | 0 |
Every winner is a simple strategy that trains on all the data or on random subsets (RS2 and Full are defined in item 3 below).
The revised conclusion now gives four recommendations for users:
- Default to a simple strategy: full-data training for fewer epochs (Full) or repeated random sampling (RS2).
- Spend the time a selector would take on more training epochs.
- If you must select a subset, keep most of the data. At every budget the best sophisticated cell is at f = 0.7, the largest subset in our grid.
- Measure the scan cost on your own dataset. For most selectors, selection time is almost entirely one full-data scan (training the scoring network plus a forward pass), and its size differs by dataset.
2. Hardware and the T_sel / T_train ratio
The ratio is the right quantity to check, so we measured it on a second server with a different GPU model. We repeated the timing probes of the six audited selectors on CIFAR-10, and the probes at two fractions on Tiny ImageNet.
The ratio barely moves: for every audited method on both datasets, selection time divided by the time of one full-data training epoch is 0.95 to 1.02 times the paper's value. With the second machine's costs, the winner, the winner's cell (f, E), and the winner's lead over the runner-up are unchanged at every budget.
On Tensor Cores: most of selection time is forward and backward passes through the same ResNet-18 that downstream training uses, so both gain the same from the same hardware. Our pipeline does not use mixed precision. The parts that may not speed up are the greedy and pairwise computations. If they speed up less, selection becomes relatively more expensive on faster hardware, the direction you describe.
A new machine needs only the timing probes measured again, not the training grid.
3. The five terms
Your questions showed which sentences left the reader guessing. We take the third first, as the others build on it.
- Budget (the submission's "anchor"). A cap in seconds on selection plus training time, under which all strategies are compared. Each dataset has 8. Four are the cost of the cheapest Uniform cell at about 10, 20, 60 and 100 full-data epochs of compute (C=10, C=20, C=60, C=100 in the submission). Four are the 10th, 50th and 90th percentiles and the maximum of all cell costs in the grid (q10, q50, q90, max). On CIFAR-10 they span 64 s to 913 s.
- Most favorable operating point. For a given budget and method, the cell (f, E) with the highest three-seed mean accuracy among those whose total cost fits the budget. Each method is represented by this cell, so none is held to a fraction or training length it would not choose.
- First-class strategy. RS2 competes at every budget under the same rule as every other method, not as a side reference baseline. The revision drops the phrase and states the comparison directly.
- RS2 and Full. RS2 draws a new random fN of the N samples independently every epoch, the with-replacement variant in the RS2 paper. Full trains on all N samples for E epochs, with the cosine schedule compressed to E epochs. RS2's Section 3 notes that RS2 without replacement "can also be viewed as training on the full dataset for fewer epochs", with an adjusted learning rate.
- Section 4.5. A cost model in which the coreset is reused: it is selected once, then used for K independent training runs with the same 200-epoch recipe. It contains no duplicated samples. Selection is paid once and spread over the K runs.
In the revision, every term is defined at first use, budgets are named only by their seconds, and the sentences you quoted have been rewritten.
We believe these changes address the concerns behind your rating, and we ask you to reconsider your score.
Follow-up to Reviewer ArUz
Dear Reviewer ArUz,
The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.
In brief: the revision credits RS2 for including selection time, and Figure 1, Related Work and the contribution list are rewritten accordingly. What we add is a comparison RS2 did not run: each method chooses its own fraction and training length under a budget in seconds, and no sophisticated selector wins any budget. On hardware, we repeated the timing on a second server with a different GPU model: the ratio of selection time to training time is 0.95 to 1.02 times the paper's value, and the winner at every budget is unchanged.
If anything remains unclear or unresolved, please let us know and we will respond before the deadline.
Thank you again for your time.
Reply to Reviewer ArUz (1/2)
Thank you for pointing out that our framing understated RS2's contribution. You are right that RS2 already includes selection time in its time-to-accuracy evaluation, and also studies dataset distillation, noisy datasets, and GPT-2 pretraining. Those latter topics are outside our scope, and our paper does not introduce budget-aware evaluation. The problem was how we described the prior work. We have redrawn Figure 1, which no longer contrasts a "conventional" evaluation with this work and now shows only the comparison we add, revised the sentence "Data Diet, DeepCore, and DRoP report accuracy at fixed fractions and treat selection as free" to identify RS2 as an exception, and revised the contribution list to state RS2's contribution.
What this paper adds beyond RS2
Our conclusion agrees with RS2's. We add a comparison that RS2 did not run, and a result that needs this comparison to be seen.
Each method chooses its own fraction and training length. In RS2, every method at a given ratio trains for the same number of rounds as full-data training (Sections 2 and 5.1, Tables 10 to 14), and selection time is added on top of that training. We let each method choose its fraction and training length within the same total seconds. A budget is a cap in seconds on selection plus training time. A cell (f, E) is a complete run on fraction f for E epochs, with the cosine schedule compressed to E. Each method enters a budget with its most accurate affordable cell. This is the comparison most favorable to sophisticated selectors: it removes the argument that they would do better at a fraction and training length they choose. We compare at eight budgets each on CIFAR-10 and Tiny ImageNet, and no sophisticated one-shot selector wins any. Across the two datasets, full-data training for fewer epochs (Full) wins 8, RS2 wins 6 and class-balanced random sampling wins 2. Our RS2 cells draw a new random fN of the N samples independently every epoch, the with-replacement variant in the RS2 paper.
The best method at a fixed fraction is not the best user of a fixed budget. After we fixed a sign error in the Herding implementation inherited from DeepCore, Herding has the best average rank at fixed fractions (2.79 over five datasets, all fractions below 1, three seeds). It wins no budget. A ranking at fixed fractions does not predict the outcome under a budget.
Why. The selection bill does not shrink with the fraction. RS2's Table 8 already shows this on CIFAR-10 for most methods, and RS2 gives the reason: they first pretrain an auxiliary model on the full dataset. On four datasets, including ImageNet-1K, one full-data scan that does not depend on the fraction (training the scoring network plus one forward pass) is 95.3% to 99.8% of Herding's selection time. Keeping less data saves no selection time and only removes training data. The price of the scan differs by dataset (one scoring epoch costs about 1.0 full-data training epoch on Tiny ImageNet and 1.2 to 1.5 on CUB-200 and ImageNet-1K), so it has to be measured on the target dataset.
Sophisticated selectors come closest when they discard the least. At every budget they can afford, their best cell is at f = 0.7, the largest subset in our grid. Held to f ≤ 0.1, they trail the winner by more than 20 pp on CIFAR-10 and 27 pp on Tiny ImageNet. At their best cell the gap is 0.2 to 5.2 pp, and they win no budget. This agrees with RS2's motivation for repeated sampling, which "maximizes overall data coverage by periodically resampling the subset" (Appendix A), and extends it to the selectors themselves. Any two subsets that each hold 70% of the data share at least 57% of their samples, so this is where the choice of subset matters least, while the scoring time is paid in full.
Stability. A simple strategy wins every budget when cells are chosen on one half of the test set and scored on the other (at least 98% of 2,000 splits), under ±10% cost perturbations, and on a second server. Selectors scored with a ResNet-20 network win no budget. The recent selectors DUAL, InfoMax and SAGE win no budget on either dataset. Reusing one subset for K training runs does not make a sophisticated selector cheaper than random selection.
Resources. We fixed nine correctness issues inherited from DeepCore and release per-run results and selected-sample indices for all 1,674 runs, and now also for the recent selectors a reviewer named. Anyone who needs a fixed subset can use these indices directly.
Sorscher et al.
You are right that the observation about Forgetting often ranking first at f ≥ 0.5 is consistent with Sorscher et al. (2022), "Beyond Neural Scaling Laws." We have added the citation at that statement and recast the observation as a consistency check, not as a contribution.