acceptodds
Withdrawn from ICLR 2027

Rethinking Sample Utility for Coreset Selection Under Extreme Compression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Coreset selection provides a practical alternative to dataset distillation by pre- serving real, inspectable samples, yet existing methods degrade sharply under extreme compression. We attribute this failure to two mismatches: criteria de- signed for larger budgets undervalue prototypical samples, while fixed evaluators cannot capture how sample utility changes with the evolving coreset. We pro- pose Model-State-Conditioned Coreset Selection (MCCS), which initializes with prototypical samples, updates utility scores using a model trained on the current subset, and filters already-learned samples to prioritize complementary informa- tion. Across CIFAR-10 and ImageNet-1K, MCCS achieves the best or tied-best coreset performance at every budget from 1 to 200 images per class. MCCS also surpasses dataset distillation as the budget or dataset complexity increases and transfers effectively across downstream tasks and architectures. These results es- tablish model-state-conditioned utility estimation as a key requirement for coreset selection under extreme compression.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
coreset selection, dataset distillation, deep learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XdHmOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an adaptive coreset selection method that estimates sample utility using a model trained on the evolving subset. It begins with representative samples, then repeatedly excludes correctly classified candidates and selects the lowest-loss examples among those still misclassified. Experiments report improvements over existing coreset methods, and competitive performance against dataset distillation.

Critical strengths
  • Consistent empirical improvements
  • Ablations provide meaningful support for the proposed design
  • The idea of evolving subsets is interesting
Critical weaknesses
  • The wall-cock selection time is occluding that the MCSS is probably the slowest among core-set selection methods and also not generally faster than dataset distillation (for instance, RDED just selects 4 random images and pack them into one). A more nuanced claim would be appropriate.
  • The authors acknowledge that DQAS already retrains its evaluator on the growing subset. Consequently, model-dependent selection itself is not new and the contribution is primarily the combination of representative initialization, filtering, and low-loss selection. The paper could establish more clearly what this combination teaches beyond a successful empirical recipe.
  • Table 4 should also compare against random initialization.
Questions for authors
  • Could the authors add other coreset selection methods for time comparison (i.e., using DeepCore library)? Also, RDED should be included as fast dataset distillation method.
  • The paper discusses coverage-based coreset selection methods but never tests them, would it be possible to compare with such, for example SubZeroCore?
  • Regarding Appendix D, could the authors provide a pairwise cosine similarity measure on the embeddings of the selected images to showcase their diversity claim?
  • Did it occur to the authors to try the opposite direction, shrinking step-by-step the subset?
Reviewer_YAZrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Model-State-Conditioned Coreset Selection (MCCS) for selecting real training samples under extreme compression. MCCS initializes the coreset with representative samples, then iteratively trains a model on the accumulated subset, filters out unselected samples that the current model classifies correctly, and ranks the remaining candidates using negative cross-entropy.

Experiments on CIFAR-10 and ImageNet-1K cover budgets from 1 to 200 images per class (IPC), with comparisons against coreset selection and dataset distillation methods. Component ablations, stage-count experiments, downstream transfer, and cross-architecture evaluations further examine the method's effectiveness and generalization.

Critical strengths
  1. Strong empirical evaluation. MCCS is evaluated on CIFAR-10 and ImageNet-1K across a broad range of compression budgets, with results reported as means and standard deviations over three runs. It achieves 63.2% accuracy on CIFAR-10 at 50 IPC and 77.5% at 200 IPC, demonstrating competitive performance against existing coreset selection methods.

  2. Informative component ablations and broader evaluation. Table 4 shows substantial performance degradation when learned-concept filtering is removed or the evaluator is fixed. At 200 IPC, removing filtering reduces accuracy by 12.4 percentage points on CIFAR-10 and 14.9 points on ImageNet-1K. Additional downstream-transfer and cross-architecture experiments provide further evidence of the method's practical effectiveness.

Critical weaknesses
  1. Limited methodological novelty. MCCS combines representative-sample initialization, iterative model retraining, filtering of correctly classified candidates, and loss-based ranking. Although this combination is empirically effective, the method primarily integrates familiar selection strategies rather than introducing a clearly distinct algorithmic mechanism. The paper does not sufficiently establish that this combination constitutes a substantial methodological advance, which limits the significance of its contribution.

  2. The central utility-estimation claim lacks direct validation. MCCS uses negative cross-entropy under the current coreset-trained model as a proxy for marginal sample utility, but the experiments only show that this scoring strategy improves final accuracy. They do not establish whether the score correlates with the actual marginal validation-performance gain from adding a candidate. Thus, the results support the effectiveness of the selection heuristic, but not the claim that it accurately estimates marginal utility.

Questions for authors
  1. Figure 1 examines the effect of different numbers of selection stages (T) on performance, while Appendix Tables 5–6 report the stage configurations used for different budgets. Could the authors clarify whether the number of stages was selected using a data split independent of the final performance evaluation? This would help clarify the tuning protocol and reproducibility of the method.

  2. Could the authors provide direct evidence of the relationship between the proposed negative-loss score and actual marginal sample utility? For example, on a manageable subset of candidates, could they measure the change in validation performance after adding individual samples or small batches and compare these gains with the proposed score rankings? This would help distinguish an effective selection heuristic from a score that accurately estimates marginal utility.

Reviewer_9d1UOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at why coreset selection falls apart at very low budgets (1–200 images per class) and argues the problem is two mismatches: criteria that over-value hard samples when you actually need prototypes first, and a fixed scorer that can't track what the coreset has already learned. The proposed MCCS starts from prototypical samples, re-scores candidates with a model trained on the current subset, and drops samples the model already gets right. It beats other coreset methods at essentially every budget on CIFAR-10 and ImageNet-1K, and overtakes dataset distillation once the budget gets larger.

Critical strengths

The framing is clean and the diagnosis is convincing — the budget-mismatch and fixed-evaluator analyses (Tables 1 and 2) make the motivation feel earned rather than post-hoc. The ablations are thorough, the three components are each shown to matter, and the comparison against distillation under a fair hard-label storage budget is a nice touch that a lot of this literature gets wrong.

Critical weaknesses

The gains over the best coreset baselines are fairly small (1–3pp) and a lot of the "wins" over distillation are really just "distillation is bad on cross-architecture transfer," which somewhat inflates the story. The method also quietly relies on a teacher trained on the full dataset for initialization, which undercuts the efficiency pitch a little. Finally, the number-of-stages hyperparameter is tuned per dataset/budget, and I'd like to see how sensitive the results are to that.

Questions for authors

The gains over the best coreset baselines are fairly small (1–3pp) and a lot of the "wins" over distillation are really just "distillation is bad on cross-architecture transfer," which somewhat inflates the story. The method also quietly relies on a teacher trained on the full dataset for initialization, which undercuts the efficiency pitch a little. Finally, the number-of-stages hyperparameter is tuned per dataset/budget, and I'd like to see how sensitive the results are to that.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.