acceptodds
Under review as a conference paper at ICLR 2027

Are Coreset Selection Methods Worth Their Cost?

Abstract

Data selection methods, often called coreset selection, pick a training subset to make training cheaper. The usual evaluation reports downstream accuracy at a fixed subset size, which leaves two quantities out of the metric: the time spent selecting the subset, and the training recipe behind each reported number. We close that gap with an end-to-end benchmark that standardizes downstream training and prices selection and training on one wall-clock axis estimated from controlled timing probes, spanning 4 datasets from CIFAR-10 to ImageNet-1K, 11 selectors, 5 fractions, and 3 seeds, across 1,674 released runs with their selected indices. Repeated-sampling work has shown that budget-aware evaluation already favors random strategies, and our two budget studies test whether that holds when every selector is granted its most favorable tested operating point. Across eight budget anchors on each of CIFAR-10 and Tiny ImageNet, no anchor is won by a sophisticated selector: every winner is class-balanced random sampling, repeated random sampling, or full-data training. In nearby-cost comparisons on ImageNet-1K, training on all data for fewer epochs has the highest accuracy among the operating points we probe. A per-dataset cost audit shows that selection cost is dominated at every scale by a fixed full-dataset scan, so it cannot be amortized away by selecting a smaller fraction, and its absolute size does not extrapolate from one dataset to another. We further separate cost amortization at a fixed architecture from cross-architecture accuracy transfer, and document 9 correctness fixes to a widely used codebase, one of which shifts a standard Herding baseline by 5.9 percentage points (pp). Selection time is not free preprocessing, and an evaluation that ignores it measures the wrong quantity.

Primary area:
datasets and benchmarks
Keywords:
coreset selection, data pruning, benchmark, wall-clock budget, cost audit, ImageNet, reproducibility

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ArUzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper evaluates whether coreset selection methods are worth their cost. Previous works have treated the preprocessing cost as “free”. This paper tries to standardize downstream training and prices selection on a wall-clock axis that is estimated from timing probes. Results show that selection cost is dominated by a fixed full-dataset scan.

Critical strengths
  • The paper is well motivated and easy to follow.
  • Paper tries to provide practical insights, also it is appreciated to fix bugs as reported that was done in previous works. This is good for the community in general.
Critical weaknesses
  • Unclear contribution beyond prior work: RS2 already accounts for selection time and evaluates dataset distillation, noisy datasets, and GPT-2 retraining. The finding about hard examples at larger fractions also appears in prior work. The paper should clearly explain what it adds beyond these results.

  • Unclear practical takeaway and dependence on hardware: The paper does not clearly explain what its results tell users about which selection method to choose or what are the insights

Questions for authors

Clarity

Could you clarify the following statements?

“granting each selector its most favorable operating point under one recipe” — What is the most favorable operating point?

“We include RS2 as a first-class strategy” — What is a first-class strategy?

“Selection cost settles the cheapest anchors before accuracy enters” — What is an anchor?

“These compressed full-data points are the strategy equivalent to RS2 without replacement, listed as Full” — This is confusing. What do RS2 and Full refer to in the paper?

In Section 4.5, it is unclear what setting is being evaluated. Is the coreset being reused? If so, does this result in multiple instances of the same data point?

Soundness

The statement “Data Diet, DeepCore, and DRoP report accuracy at fixed fractions and treat selection as free,” Figure 1, and the contribution claiming that the paper introduces measuring preprocessing time are problematic: RS2 does not treat selection as free; this has already been done in that paper. Furthermore, RS2 also includes dataset distillation, noisy datasets (robustness analysis), and GPT-2 retraining. Therefore, it is unclear what the main contribution of the paper is.

What is the practical takeaway from the experiments in the paper? Is it that no single method wins in all settings? Or that all of these coreset selection methods incur unwanted costs?

Doesn’t the ratio of T_sel to T_train depend on the hardware used? Using a newer GPU could favor Random, as T_train can be reduced with newer hardware, such as Tensor Cores. However, coreset selection methods might not utilize the new hardware as fully as training does. Wouldn’t this make the ratio and the paper’s conclusions dependent on the selected hardware?

“At large fractions hard-example scoring often leads and the methods converge. Forgetting often ranks first at f≥0.5, which matches hard-to-learn examples becoming more useful at larger fractions.” — Isn’t this already shown in Sorscher et al.’s “Beyond Neural Scaling Laws”? Both theoretical claims and experimental evidence for this already exist. Does this result simply confirm that the finding holds in a different setting (model, dataset)?

6 comments
AuthorsOpenReview ↗

Reply to Reviewer ArUz (1/2)

Thank you for pointing out that our framing understated RS2's contribution. You are right that RS2 already includes selection time in its time-to-accuracy evaluation, and also studies dataset distillation, noisy datasets, and GPT-2 pretraining. Those latter topics are outside our scope, and our paper does not introduce budget-aware evaluation. The problem was how we described the prior work. We have redrawn Figure 1, which no longer contrasts a "conventional" evaluation with this work and now shows only the comparison we add, revised the sentence "Data Diet, DeepCore, and DRoP report accuracy at fixed fractions and treat selection as free" to identify RS2 as an exception, and revised the contribution list to state RS2's contribution.

What this paper adds beyond RS2

Our conclusion agrees with RS2's. We add a comparison that RS2 did not run, and a result that needs this comparison to be seen.

Each method chooses its own fraction and training length. In RS2, every method at a given ratio trains for the same number of rounds as full-data training (Sections 2 and 5.1, Tables 10 to 14), and selection time is added on top of that training. We let each method choose its fraction and training length within the same total seconds. A budget is a cap in seconds on selection plus training time. A cell (f, E) is a complete run on fraction f for E epochs, with the cosine schedule compressed to E. Each method enters a budget with its most accurate affordable cell. This is the comparison most favorable to sophisticated selectors: it removes the argument that they would do better at a fraction and training length they choose. We compare at eight budgets each on CIFAR-10 and Tiny ImageNet, and no sophisticated one-shot selector wins any. Across the two datasets, full-data training for fewer epochs (Full) wins 8, RS2 wins 6 and class-balanced random sampling wins 2. Our RS2 cells draw a new random fN of the N samples independently every epoch, the with-replacement variant in the RS2 paper.

The best method at a fixed fraction is not the best user of a fixed budget. After we fixed a sign error in the Herding implementation inherited from DeepCore, Herding has the best average rank at fixed fractions (2.79 over five datasets, all fractions below 1, three seeds). It wins no budget. A ranking at fixed fractions does not predict the outcome under a budget.

Why. The selection bill does not shrink with the fraction. RS2's Table 8 already shows this on CIFAR-10 for most methods, and RS2 gives the reason: they first pretrain an auxiliary model on the full dataset. On four datasets, including ImageNet-1K, one full-data scan that does not depend on the fraction (training the scoring network plus one forward pass) is 95.3% to 99.8% of Herding's selection time. Keeping less data saves no selection time and only removes training data. The price of the scan differs by dataset (one scoring epoch costs about 1.0 full-data training epoch on Tiny ImageNet and 1.2 to 1.5 on CUB-200 and ImageNet-1K), so it has to be measured on the target dataset.

Sophisticated selectors come closest when they discard the least. At every budget they can afford, their best cell is at f = 0.7, the largest subset in our grid. Held to f ≤ 0.1, they trail the winner by more than 20 pp on CIFAR-10 and 27 pp on Tiny ImageNet. At their best cell the gap is 0.2 to 5.2 pp, and they win no budget. This agrees with RS2's motivation for repeated sampling, which "maximizes overall data coverage by periodically resampling the subset" (Appendix A), and extends it to the selectors themselves. Any two subsets that each hold 70% of the data share at least 57% of their samples, so this is where the choice of subset matters least, while the scoring time is paid in full.

Stability. A simple strategy wins every budget when cells are chosen on one half of the test set and scored on the other (at least 98% of 2,000 splits), under ±10% cost perturbations, and on a second server. Selectors scored with a ResNet-20 network win no budget. The recent selectors DUAL, InfoMax and SAGE win no budget on either dataset. Reusing one subset for K training runs does not make a sophisticated selector cheaper than random selection.

Resources. We fixed nine correctness issues inherited from DeepCore and release per-run results and selected-sample indices for all 1,674 runs, and now also for the recent selectors a reviewer named. Anyone who needs a fixed subset can use these indices directly.

Sorscher et al.

You are right that the observation about Forgetting often ranking first at f ≥ 0.5 is consistent with Sorscher et al. (2022), "Beyond Neural Scaling Laws." We have added the citation at that statement and recast the observation as a consistency check, not as a contribution.

AuthorsOpenReview ↗

Reply to Reviewer ArUz (2/2)

This reply covers the practical takeaway, hardware, and the five terms. Reply 1/2 covered RS2 and Sorscher et al.

1. Practical takeaway

In the settings we evaluate, sophisticated one-shot selectors should not be the default: the accuracy they buy with selection time is smaller than what the same time buys when spent on training. Of your two readings, this is closer to "all of these coreset selection methods incur unwanted costs" than to "no single method wins in all settings", and it is more specific. At none of the budgets we compare does a sophisticated selector come out ahead.

The evidence is the comparison under 8 budgets each on CIFAR-10 and Tiny ImageNet (caps in seconds on selection plus training time, defined in item 3 below). Each method enters with the best cell (fraction f, E epochs) it can afford.

WinnerBudgets wonWhere
Full (all data, fewer epochs)8CIFAR-10: 5, Tiny ImageNet: 3
RS26CIFAR-10: 913 s, Tiny ImageNet: 5
Uniform (class-balanced random)2CIFAR-10: 64 s and 78 s
Any sophisticated selector0

Every winner is a simple strategy that trains on all the data or on random subsets (RS2 and Full are defined in item 3 below).

The revised conclusion now gives four recommendations for users:

  1. Default to a simple strategy: full-data training for fewer epochs (Full) or repeated random sampling (RS2).
  2. Spend the time a selector would take on more training epochs.
  3. If you must select a subset, keep most of the data. At every budget the best sophisticated cell is at f = 0.7, the largest subset in our grid.
  4. Measure the scan cost on your own dataset. For most selectors, selection time is almost entirely one full-data scan (training the scoring network plus a forward pass), and its size differs by dataset.

2. Hardware and the T_sel / T_train ratio

The ratio is the right quantity to check, so we measured it on a second server with a different GPU model. We repeated the timing probes of the six audited selectors on CIFAR-10, and the probes at two fractions on Tiny ImageNet.

The ratio barely moves: for every audited method on both datasets, selection time divided by the time of one full-data training epoch is 0.95 to 1.02 times the paper's value. With the second machine's costs, the winner, the winner's cell (f, E), and the winner's lead over the runner-up are unchanged at every budget.

On Tensor Cores: most of selection time is forward and backward passes through the same ResNet-18 that downstream training uses, so both gain the same from the same hardware. Our pipeline does not use mixed precision. The parts that may not speed up are the greedy and pairwise computations. If they speed up less, selection becomes relatively more expensive on faster hardware, the direction you describe.

A new machine needs only the timing probes measured again, not the training grid.

3. The five terms

Your questions showed which sentences left the reader guessing. We take the third first, as the others build on it.

  • Budget (the submission's "anchor"). A cap in seconds on selection plus training time, under which all strategies are compared. Each dataset has 8. Four are the cost of the cheapest Uniform cell at about 10, 20, 60 and 100 full-data epochs of compute (C=10, C=20, C=60, C=100 in the submission). Four are the 10th, 50th and 90th percentiles and the maximum of all cell costs in the grid (q10, q50, q90, max). On CIFAR-10 they span 64 s to 913 s.
  • Most favorable operating point. For a given budget and method, the cell (f, E) with the highest three-seed mean accuracy among those whose total cost fits the budget. Each method is represented by this cell, so none is held to a fraction or training length it would not choose.
  • First-class strategy. RS2 competes at every budget under the same rule as every other method, not as a side reference baseline. The revision drops the phrase and states the comparison directly.
  • RS2 and Full. RS2 draws a new random fN of the N samples independently every epoch, the with-replacement variant in the RS2 paper. Full trains on all N samples for E epochs, with the cosine schedule compressed to E epochs. RS2's Section 3 notes that RS2 without replacement "can also be viewed as training on the full dataset for fewer epochs", with an adjusted learning rate.
  • Section 4.5. A cost model in which the coreset is reused: it is selected once, then used for K independent training runs with the same 200-epoch recipe. It contains no duplicated samples. Selection is paid once and spread over the K runs.

In the revision, every term is defined at first use, budgets are named only by their seconds, and the sentences you quoted have been rewritten.

We believe these changes address the concerns behind your rating, and we ask you to reconsider your score.

AuthorsOpenReview ↗

Follow-up to Reviewer ArUz

Dear Reviewer ArUz,

The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.

In brief: the revision credits RS2 for including selection time, and Figure 1, Related Work and the contribution list are rewritten accordingly. What we add is a comparison RS2 did not run: each method chooses its own fraction and training length under a budget in seconds, and no sophisticated selector wins any budget. On hardware, we repeated the timing on a second server with a different GPU model: the ratio of selection time to training time is 0.95 to 1.02 times the paper's value, and the winner at every budget is unchanged.

If anything remains unclear or unresolved, please let us know and we will respond before the deadline.

Thank you again for your time.

Reviewer_GjnYOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a benchmark that measures the total wall-clock time for both the selection process and training for data selection methods. This would allow future developers to accurately present the timing efficiency of data selection methods, not only their accuracy. The paper argues that since scanning a whole dataset takes the most amount of time, complex selection algorithms actually lose their advantages to simpler strategies when it comes to strict time budgets.

Critical strengths

The paper provides a more realistic way to test data selection methods by measuring the total time they take to run, not just their final accuracy. The paper tested many different methods across several datasets using the exact same rules, making sure the comparisons are completely fair.

Critical weaknesses

The scope of the paper is narrow because it only tests methods where data is selected exactly once before training begins, so it can't be utilized for dynamic or time-based methods that might be more efficient. All of the conclusions rely on exact time measurements taken on one specific computer setup, so a practitioner must run the complete evaluation suite to provide an accurate comparison.

Questions for authors
  1. The paper used a standard ResNet-18 model to score the data. What if one used a much smaller or pre-trained model? Would that make the selection process fast enough to beat random sampling?
  2. The paper measures time using just one computer with one GPU. Since data scoring can easily be split across many GPUs, how would using multiple GPUs change the results?
1 comment
AuthorsOpenReview ↗

Reply to Reviewer GjnY

Thank you for the review, and for questions that widened the paper's scope: we have now measured a smaller scoring network and a second machine. The Informativeness score tells us the contribution did not come through, and the revision states it more directly.

What is new. No sophisticated selector wins any budget, even though each chooses its own fraction and training length. A budget is a cap in seconds on selection plus training time, and we use eight each on CIFAR-10 and Tiny ImageNet. Fixed-fraction rankings do not predict this: with a sign error in a widely used implementation fixed, Herding ranks first on average at fixed fractions and still wins no budget.

Your questions added two results. A smaller scoring network (Q1) makes selection only 2.3 to 2.8× faster and wins no budget. On a second server (Weakness 2), each method's selection-to-training time ratio changes by a factor of 0.95 to 1.02 and every budget's winner is unchanged. Three recent methods added at another reviewer's request (DUAL, InfoMax, SAGE) win no budget either.

One-shot scope. The restriction to one-shot methods, which select once before training, is deliberate. Each one we audited pays for selection once, and that payment already outweighs its accuracy gain. The budget comparison also covers dynamic methods: RS2, which redraws a random subset every epoch at almost no selection cost, enters the same budgets with its total time and wins six of them, in line with your point that dynamic methods may be more efficient. Dynamic methods that score with statistics training already computes, such as InfoBatch, change the cost structure and are outside our scope, and we did not test them.

Dependence on one machine. The conclusions are not tied to our machine. We re-measured the audited selectors' CIFAR-10 probes and the Tiny ImageNet probes at two fractions on a second server with a different GPU model, and the cost ordering of methods is nearly the same (Spearman 0.990 for CIFAR-10 selection costs). On both datasets, for every audited method, the ratio of selection time to one full-data training epoch is 0.95 to 1.02 of the paper's value. With the second server's costs, every budget's winner is unchanged.

The reason: for these methods selection time is mostly training the scoring network (the same ResNet-18, data and data-loading pipeline as downstream training) plus one forward pass, so hardware speeds up both terms by nearly the same factor. Accuracy does not depend on hardware, so exact numbers on another machine need only the timing probes repeated, not the training suite.

Q1. A smaller or pretrained scoring network.

Smaller. We scored CIFAR-10 with ResNet-20 (Coleman et al., 2020), which needs 13.7× fewer FLOPs per image than ResNet-18. Selection time drops far less: Herding 239 s to 94 s, Forgetting 241 s to 86 s, GraNd 85 s to 37 s. The ResNet-20-scored subsets (three seeds) are also less accurate at the shortest training than ResNet-18-scored ones (70% of the data for 14 epochs: Forgetting 81.1 vs 84.4, GraNd 82.2 vs 85.0). The gap closes at the longest training (Forgetting, 143 epochs: 94.8 vs 94.5), but scoring plus training then fits only the largest budget, 913 s, and is 0.2 pp behind its winner. ResNet-20 wins no budget.

Pretrained. We tested Herding, which ranks first on average at fixed fractions, and kCenter on frozen ImageNet-pretrained features, with pretraining left uncharged. Selection, including feature extraction, takes 9 to 15 s (Herding) and 10 s (kCenter). kCenter is level with the winner at 78 s and 351 s (within its seed SD) and 0.8 to 5.0 pp behind at the other five it can afford. Herding is behind the winner at every budget it can afford (reply to BTjg Q3).

Q2. Multiple GPUs. You are right that scoring can be split across GPUs. Training can be split too, and we expect about the same speedup for both, so the comparison between methods would not change.

The reason is that most of selection time is not the scoring forward pass. It is training the scoring network, which is the same work as downstream training (same network and data). On CIFAR-10, Herding, Forgetting, kCenter and Moderate take 228 to 241 s to select on average, nearly all of it the 40 epochs of scoring-network training. GradMatch and GraphCut (Submodular in the paper) are the exceptions. Their greedy step is long (1,485 s and 432 s when keeping 70% of the data) and proceeds one step after another, which is hard to parallelize, so more GPUs help them less.

Our budgets count single-GPU seconds and so measure compute, which stays the same however many GPUs are used.

Revision: one-shot scope in the abstract, and new appendices with the ResNet-20, frozen-feature, second-server, DUAL, InfoMax and SAGE results.

Reviewer_BTjgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an end-to-end benchmark of 11 fixed-subset selection methods for supervised image classification, evaluating both selection and downstream training costs under a controlled wall-clock protocol. The experiments cover CIFAR-10, Tiny ImageNet, CUB-200, and ImageNet-1K, with additional results on Retinal OCT. The methods span random sampling, geometric selection, training-dynamics scoring, submodular optimization, and gradient matching.

Under the nominal audited costs, no sophisticated selector wins any of the eight budget anchors on either CIFAR-10 or Tiny ImageNet. In nearby-cost ImageNet-1K comparisons, reduced-epoch full-data training achieves the highest accuracy among the tested operating points. The paper also decomposes selection costs and identifies nine correctness fixes in a widely used coreset-selection codebase, including a Herding implementation error that changes reported accuracy by approximately 5.9 percentage points.

Critical strengths
  1. A practically meaningful end-to-end comparison. The paper evaluates selection methods under matched budgets that include both selection and downstream training, rather than relying solely on accuracy at fixed subset fractions. Its finding that simple strategies win all 16 nominal budget anchors on CIFAR-10 and Tiny ImageNet challenges the assumption that sophisticated selection is necessarily worthwhile. The nearby-cost ImageNet experiments further show that reduced-epoch full-data training can outperform the tested subset-selection alternatives. These results offer actionable guidance on when selection overhead may outweigh its benefits.

  2. A useful diagnosis of selection costs. The paper goes beyond method rankings to explain why selection can be inefficient. Its timing analysis attributes 95.3–99.8% of Herding's measured selection cost across the four audited datasets to a fraction-independent component. Consequently, selecting a smaller subset reduces downstream training time without proportionally reducing selection overhead. This provides a concrete explanation for the observed budget results. The implementation audit also identifies a specific source of error in existing comparisons, including a correction that materially changes the reported Herding baseline.

Critical weaknesses
  1. Test-set-driven operating-point selection weakens the headline comparisons. The fixed-fraction analysis uses peak test accuracy, thereby selecting an epoch based on test-set performance. The E1 budget analysis instead uses recorded final accuracy, but still selects the best affordable fraction-and-epoch configuration based on that accuracy. Using final rather than peak accuracy avoids one form of test-set maximization, but does not eliminate test-set-driven configuration selection. The resulting best-of-grid comparisons may be optimistically biased, particularly when the winner margins are small. The central budget conclusions should be re-evaluated with operating points selected on an independent validation set.

  2. The benchmark does not adequately cover recent selection methods relevant to its central question. The 11 methods cover several important classical families, including random sampling, geometric selection, training-dynamics scoring, submodular optimization, and gradient matching. However, the coverage is less convincing as a benchmark of contemporary methods. For example, DUAL (Cho et al., ICML 2025) [1] uses early-training difficulty and prediction uncertainty to reduce pruning costs; InfoMax (Tan et al., ICLR 2025) [2] jointly models sample information and redundancy. SAGE (Jha and Ahmadi-Asl, 2025) [3] is also relevant to efficient subset selection through streaming gradient sketches. These omissions matter because the benchmark's central question is whether selection overhead is justified by downstream accuracy. Methods specifically designed to reduce scoring costs or account for sample redundancy may have substantially different cost–accuracy trade-offs from the implemented baselines. The authors should include representative recent methods where feasible or more explicitly limit the conclusions to the evaluated method set.

[1] Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty.

[2] Data Pruning by Information Maximization.

[3] SAGE: Streaming Agreement-Driven Gradient Sketches for Representative Subset Selection.

Questions for authors
  1. Could the authors select the fraction-and-epoch configuration for each method using an independent validation set, then report test accuracy only for the selected configuration? The fixed-fraction results should likewise avoid choosing the peak epoch based on test accuracy. How would these changes affect the budget winners and their margins?

  2. How robust are the budget winners to relative errors in the estimated costs? Appendix C reports that independent ±10% perturbations to selection and training cost components produce at least one sophisticated-selector winner in 361 of 2,000 CIFAR-10 draws and 167 of 2,000 Tiny ImageNet draws, whereas common-factor perturbations preserve the nominal winners. These perturbations are not presented as a calibrated measurement-error distribution, so they do not invalidate the nominal results. Nevertheless, could the authors report per-anchor decision stability and clarify how readers should interpret the claim that no budget anchor is won by sophisticated selection?

  3. The benchmark derives learned selection signals from a ResNet-18 trained from scratch, avoiding pretrained features that could confound selection quality with transfer-learning quality. This is a reasonable control, but it excludes methods whose practical effectiveness depends on pretrained or self-supervised representations. Could the authors clarify how this design choice limits the scope of the conclusions, and whether a separate controlled experiment with frozen representations would change the relative ranking or selection costs?

  4. The primary experiments focus on conventional supervised image classification, while most selection methods use class-balanced selection. Could the authors clarify whether the conclusions are expected to hold under label noise, long-tailed class distributions, or substantial distribution shift? If additional experiments are outside the intended scope, the paper should explicitly delimit the settings to which its conclusions apply.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer BTjg (1/2)

Q1. Selecting operating points on held-out data.

Thank you for noting that choosing the best affordable (f, E) by final test accuracy still uses test data. We agree and added a held-out analysis. On CIFAR-10, we trained 26 cells again (12 simple, 14 sophisticated), with three seeds each. We split the test set into two class-stratified halves, A and B, 2,000 times. Each time, each method chose its best affordable cell on A using measured costs, and we report accuracy on B.

One rule, the same for every method, decides which cells were trained again. A method is a contender at a budget if its best affordable cell is within 1 pp of that budget's best score (0.5 pp at 653 and 913 s). For each contender we trained again its best affordable cell and every affordable cell within 0.5 pp of it.

BudgetA-half winnerSimple winsMedian B-half gap (pp)
64 sUniform (0.7, 14), 100%100%N/A
78 sUniform (0.7, 14), 98%100%N/A
132 sRS2 (0.7, 29), 63%; Full (1, 20), 37%100%N/A
351 sRS2 (0.7, 29), 61%; Full (1, 20), 36%98%0.20
393 sFull (1, 60), 91%100%3.3
653 s (two budgets)RS2 (0.7, 143), 59%; Full (1, 100), 41%100%0.73
913 sRS2 (0.5, 200), 77%99.7%0.19

No sophisticated selector in the main grid is affordable at 64, 78, or 132 s. The B-half gap is the best simple-strategy accuracy minus the best sophisticated-selector accuracy. The winner's accuracy on A exceeds its accuracy on B by 0.01–0.13 pp on CIFAR-10.

On Tiny ImageNet, the same analysis over 87 runs and 2,000 splits gives simple strategies all eight budget wins in every split. At the five budgets where sophisticated selectors are affordable, median B-half gaps are 1.3–4.8 pp. The same A-minus-B difference is at most 0.28 pp.

We also replaced peak test accuracy with last-epoch accuracy in the fixed-fraction results. The change has little effect on the rankings. Across the four natural-image datasets, last-epoch accuracy is on average 0.03–0.40 pp below peak accuracy, depending on the dataset. CUB-200's third and fourth places swap, CIFAR-10's fifth and sixth become tied, and Retinal OCT changes order within the top eight while its bottom three remain unchanged. The aggregate ranking remains Herding first and Uniform second.

Q2. Sensitivity to cost errors.

Using last-epoch accuracy, independent ±10% cost perturbations with the submission's cost estimates produce at least one sophisticated-selector winner in 332 of 2,000 CIFAR-10 draws and 172 of 2,000 Tiny ImageNet draws. The previously cited totals of 361 and 167 used peak accuracy. The table gives the number of draws in which a sophisticated selector wins each budget:

Budget (CIFAR-10 / Tiny ImageNet)64 / 451 s78 / 482 s132 / 934 s351 / 2,302 s393 / 2,769 s653 / 4,605 or 4,606 s913 / 6,445 s
CIFAR-10, submission costs22903318600
Tiny ImageNet, submission costs1040720000
Updated costs0000000

Every win in the first two rows is Submodular. The submission charged Submodular, Craig, and GradMatch no selection time, the most favorable assumption for them. Even so, they won no budget with unperturbed costs, and perturbing the costs gave Submodular the wins above.

We have now measured the three methods' selection times on CIFAR-10: 335–432 s for Submodular, 302–317 s for Craig, and 245–1,485 s for GradMatch, each above Herding's at the same fraction (234–245 s). On Tiny ImageNet we charge a lower bound: each of the three first trains its scoring network for 40 full-data epochs (about 1,820 s), and we charge only that. With these costs, no sophisticated selector wins any budget in any draw on either dataset (last row). This does not depend on the exact charge: on Tiny ImageNet, charging only a quarter of it (10 of the 40 epochs) still gives no sophisticated winner in any draw.

The three cheapest budgets are decided by affordability and the other five by accuracy. At those five, the best sophisticated selector on the CIFAR-10 main grid trails the winner by 0.2–3.2 pp.

We have added an appendix describing the held-out analysis and cell-inclusion rule. We have recomputed the budget tables (Tables 5 and 6) and Figure 2a using measured costs, and included measured selection times and the cost-perturbation analysis in the appendix.

AuthorsOpenReview ↗

Reply to Reviewer BTjg (2/2)

Q3. Frozen pretrained features.

We agree that from-scratch scoring excludes methods relying on pretrained or self-supervised representations. This was an intentional control, but it limits our conclusions. To test this, we added a CIFAR-10 experiment using frozen ImageNet-supervised ResNet-18 features for Herding (first on average in our fixed-fraction benchmark) and kCenter.

Even with pretraining cost not counted, neither method does better than match the winner at any budget. Including feature extraction, selection takes 9 to 15 s for Herding and 10 s for kCenter (1.6 to 2.5 full-data epochs). kCenter is level with the winner at 78 s and 351 s (within its seed SD) and 0.8 to 5.0 pp behind at the other five it can afford. Herding is behind the winner at every budget it can afford. The experiment is outside the main grid because ImageNet-1K is one of our benchmark datasets, so ImageNet features would mix selection quality with transfer.

Q4. Label noise, long tails, and distribution shift.

We tested none of these three settings, and the revised Limitations now states this.

The cost side does not depend on them. Selection time is mostly full-data training of the scoring network plus one forward pass, paid whatever the label quality or class balance.

Under label noise, we expect repeated random sampling to hold up well. A random draw ignores sample content, so each subset has the full set's share of mislabeled samples. In the RS2 paper (Okanovic et al., ICLR 2024), RS2 is the most accurate compared method at all three levels of random CIFAR-10 label flips. A score that keeps the hardest samples cannot tell hard from mislabeled, as both have high loss. This is consistent with Data Diet (Paul et al., 2021): dropping a small share of highest-EL2N samples raises accuracy, more so with 10% of CIFAR-10 labels shuffled. We did not test a selector that removes mislabeled samples, which might beat random sampling under heavy noise.

On class imbalance, we have mildly imbalanced Retinal OCT (about 4:1). Its ranking splits as on the natural-image datasets: coverage-based and random methods rank above hard-example scoring (at 5% of the data, Random 98.0% vs. EL2N 16.8%).

Recent methods: DUAL, InfoMax, and SAGE.

Thank you for naming these methods. We implemented all three on CIFAR-10 and Tiny ImageNet (three seeds each), with the same eight budgets per dataset and operating-point rule as part 1, scored by mean last-epoch test accuracy under our common recipe. None wins any budget.

MethodCIFAR-10Tiny ImageNet
DUAL5/8 affordable; within its seed SD of the winner at 351 s and 913 s (0.80 and 0.28 pp); trails by 0.81 pp at both 653 s budgets and 3.98 pp at 393 s5/8 affordable; trails by 0.8–2.8 pp
InfoMax5/8 affordable; trails by 2.2–5.4 pp5/8 affordable; trails by 3.8–5.8 pp
SAGE6/8 affordable; trails by 2.8–12.3 pp6/8 affordable; trails by 4.8–8.8 pp

Our cost accounting favors these methods. DUAL's CIFAR-10 scoring time is measured at 177 s. Elsewhere we charge only lower bounds: 30 scoring epochs for DUAL and InfoMax (about 1,338 s on Tiny ImageNet) and three full-data epochs for SAGE, one for training and two for its two training-set passes (about 18 s on CIFAR-10, 134 s on Tiny ImageNet). InfoMax feature extraction and solving and SAGE's gradient sketch are not charged, so actual costs are higher. For all three, the best affordable cell is always at f = 0.7.

We ran the full DUAL method, including its Beta sampling step, scored from the first 30 full-data epochs. In the DUAL paper's reported setting (70% of the data, 200 epochs) with its batch size and learning-rate floor, we get 95.30 ± 0.10%, close to the reported 95.51%. With our recipe, DUAL beats a static random subset in that setting (94.94 ± 0.36% vs. 94.05%). With scoring, however, it costs 1,090 s, above the largest budget (913 s).

InfoMax uses the public solver on features and scores from the same 30-epoch training as DUAL. It is below same-cell RS2 in every tested cell on both datasets. One likely reason is that its subsets favor easier examples, and dropping hard ones can reduce accuracy when 30% to 70% of the data is kept.

The revised manuscript adds appendices for frozen features and the three recent methods. We updated Related Work, released the methods and selected sample indices with the code, stated the from-scratch limitation in Limitations, and narrowed Section 3's scope to the evaluated methods.

We believe these experiments address the concerns behind your rating and leave the main conclusions unchanged. We would be grateful if you would reconsider your score in light of them.

AuthorsOpenReview ↗

Follow-up to Reviewer BTjg

Dear Reviewer BTjg,

The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.

In brief: when each method's fraction-and-epoch configuration is chosen on one half of the test set and scored on the other half, a simple strategy wins every budget in at least 98% of 2,000 splits on CIFAR-10 and in all of them on Tiny ImageNet. DUAL, InfoMax and SAGE, the methods you named, are now run on both datasets, and none wins a budget.

If anything remains unclear or unresolved, please let us know and we will respond before the deadline.

Thank you again for your time.

AuthorsOpenReview ↗

Reminder to Reviewer BTjg

Dear Reviewer BTjg,

A brief reminder that the discussion period closes on Nov 18 (AoE), about a day from now.

The two results that answer your Critical Weaknesses are the held-out choice of each method's configuration and the DUAL, InfoMax and SAGE runs on both datasets. Both are summarized in our response above. We would be glad to hear whether they resolve your concerns, and we can still answer any remaining question before the deadline.

Your comments led to the held-out analysis and the three added methods, and the paper is stronger for both. Thank you.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers. Four concerns recur: choices made on the test set, recent and cheaper selectors, the contribution beyond RS2, and reliance on one machine. We ran new experiments and uploaded the revised paper. The conclusion holds and is now stated more precisely: the accuracy a sophisticated selector buys is smaller than the accuracy the same time buys when spent on training. A budget is a cap in seconds on selection plus training time. Under each budget, every method uses its most accurate cell (a fraction and a training length) that fits, and the method with the highest accuracy wins that budget. Each dataset has eight budgets, from about ten full-data epochs of compute to the costliest grid cell.

QuestionWhat we ranResult
Test-set choices (BTjg)Held-out choice: 26 CIFAR-10 cells × 3 seeds, 87 Tiny ImageNet runsA simple strategy wins each budget in ≥98% of splits
Cost error (BTjg)Measured three selectors' selection timeNo sophisticated winner in 2,000 ±10% cost draws
Recent selectors (BTjg)DUAL, InfoMax, SAGE, both datasetsNone wins a budget
Smaller scoring network (GjnY)ResNet-20 scoring2.3 to 2.8× faster, wins no budget
Pretrained features (BTjg, GjnY)Frozen ImageNet featureskCenter level with the winner at two budgets, behind at the other five it can afford
One machine (GjnY, ArUz)Probes on a second serverCost ratios 0.95 to 1.02, same winners

Held-out operating points (BTjg Q1). Choosing each method's best cell on the test set can make the comparison optimistic. We now choose each method's best affordable cell on one class-stratified half of the test set and score it on the other, over 2,000 splits. A simple strategy (full-data training for fewer epochs, RS2, or class-balanced random sampling) wins every CIFAR-10 budget in at least 98% of splits and every Tiny ImageNet budget in all splits. The winner's optimism from that choice averages at most 0.28 pp.

Cost error (BTjg Q2). Submodular, Craig and GradMatch are now priced at measured selection time on CIFAR-10 and at a lower bound on Tiny ImageNet. None of 2,000 ±10% cost perturbations on either dataset then gives a sophisticated selector a budget.

Recent and cheaper selectors (BTjg, GjnY).

  1. DUAL, InfoMax and SAGE on both datasets, three seeds. None wins any budget. At affordable budgets, DUAL is within seed SD of the winner at two CIFAR-10 budgets and trails by 0.8 to 4.0 pp at the rest, InfoMax trails by 2.2 to 5.8 pp and SAGE by 2.8 to 12.3 pp. For all three the best cell is at f = 0.7.

  2. A ResNet-20 scoring network. It needs 13.7× fewer FLOPs per image but selects only 2.3 to 2.8× faster. Its subsets reach lower accuracy under short training, and it wins no budget.

  3. Frozen pretrained features. Herding and kCenter select on frozen ImageNet-pretrained ResNet-18 features, with pretraining not counted in the cost. kCenter is level with the winner at the 78 s and 351 s budgets (within its seed SD) and 0.8 to 5.0 pp behind at the other five it can afford. Herding is behind the winner at every budget it can afford.

What the paper adds to RS2 (ArUz). Our conclusion agrees with RS2's, and we add a comparison RS2 did not run. In RS2, every method trains for as many epochs as full-data training, with selection time added on top. We let each method choose its own fraction and training length under each budget. This is the comparison most favorable to sophisticated selectors, and none wins any budget. Across the two datasets, full-data training for fewer epochs wins 8 budgets, RS2 wins 6 and class-balanced random sampling wins 2. Herding, with its sign error fixed, ranks first on average at fixed fractions and wins no budget, so the ranking at fixed fractions does not predict the outcome under a budget.

One machine (GjnY, ArUz). On a second server with a different GPU model, we repeated the audited selectors' CIFAR-10 probes and the Tiny ImageNet probes at two fractions. For every audited method, the ratio of selection time to one full-data training epoch is 0.95 to 1.02 of the paper's value, and every budget's winner is unchanged. New hardware needs only the timing probes repeated, not the training runs.

Revisions (uploaded). (i) Related Work and the contribution list credit RS2, Section 4.3 credits Sorscher et al. for the hard-example result, and Figure 1 is redrawn. (ii) New appendices cover the six experiments in the table. (iii) Fixed-fraction tables report last-epoch accuracy. (iv) Submodular, Craig and GradMatch are charged selection time, budgets are labeled in seconds, and ArUz's terms are defined at first use. (v) The conclusion gives four practitioner rules. (vi) Abstract, Section 3 and Limitations state the scope: one-shot selection, clean-label image classification, the evaluated methods, from-scratch scoring.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.