acceptodds
Under review as a conference paper at ICLR 2027

Measure, Don't Prescribe: Inferring Budget-Dependent Difficulty Preference for Dataset Pruning

Abstract

Dataset pruning selects a compact subset to reduce the cost of training on large datasets. Which examples are worth keeping depends on pruning budget: small budgets favor easier, representative examples, whereas larger budgets increasingly favor harder, more informative examples. Existing methods first prescribe a parametric form for the difficulty preference (e.g., hard cutoff, Beta distribution, or contiguous window), and adapt across budgets by adjusting its parameters (e.g., a cutoff ratio, shape parameters, or window position). However, identifying the optimal parameter values at the target budget requires trial-and-error search, and the form is fixed in advance, which may not suit every setting. Instead, we propose Difficulty Preference Scoring (DPS), which replaces prescribe-then-search with probe-then-infer: DPS first constructs a difficulty-balanced probing subset, then extracts from its training dynamics the signals that determine how the budget should be adjusted across difficulty levels, and corrects the allocation accordingly. This reduces the adaptation cost at a target budget from extensive trial and error to a single probing run, and lets the preference be inferred from the target setting rather than prescribed in advance. Across a wide range of budgets (1%–50%) and diverse datasets (CIFAR-10, CIFAR-100, QNLI, ImageNet and SQuAD 1.1), DPS achieves superior performance over state-of-the-art pruning methods.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Dataset Pruning, Data Efficiency, Coreset Selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zGS6OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a dataset pruning method. First the difficulty of each sample is evaluated by AUM or variants. Each difficulty level (there are 50 or so such levels) is initially pruned at the same level depending on the budget. Then these parts are probed and a new pruning levels are constructed according to their notion of "excess fitting" by comparing the trained full model's accuracy and the checkpoint's accuracy.

Critical strengths

The paper develops some interesting ideas like excess fitting on dataset pruning that may be applicable in different context. The paper is generally well written.

Critical weaknesses
  • First training a model to assess difficulty, then not stopping there are retraining full using parts of the model implies enormous computational complexity and defeats the purpose of dataset pruning. The idea of dataset pruning is that one already has a huge dataset and want to "quickly" get rid of irrelevant parts to speed up overall training. Assessing difficulty after full training (and actually doing this again for the probing subsets) does not seem to me an interesting problem. In the paper, the argument is that this is done "once" and then applied to other downstreams tasks but I do not find this convincing as every downstream tasks would have its own notion of easy and difficult. This is the major issue with the paper.
  • The sentence "Existing difficulty scores are budget-agnostic: they reflect each example’s contribution to learning under full-data training, which may differ at the target budget." did not really make sense to me despite multiple times reading it and I do not understand the intention. This claim of existing algos are budget agnostic is also repeated multiple times... In fact, all algos work with some concept of a budget (e.g. in the form of a threshold in the simplest case, threshold low means budget high or threshold high means budget low). Also, in their simulations they have previous work under the umbrella of "budgeted method" which seems to contradict this claim.
  • There has been other recent work on dataset pruning that the paper does not elaborate or compare to. The reference list seems limited in this context.
  • The theorems seem to be thrown around for the sake of being theorems and do not establish the critical facts, e.g. why this scheme is supposed to be better than the others.
Questions for authors

No questions.

Reviewer_eV3vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Difficulty Preference Scoring (DPS), a framework for budget-adaptive dataset pruning that infers the preferred allocation across difficulty levels from a single probing run rather than prescribing a parametric preference and searching for its optimal parameters. DPS combines two signals extracted from training dynamics: budget-conditioned difficulty and excess fitting. Their product is used to derive a sampling distribution over difficulty strata. Experiments on CIFAR-10, CIFAR-100, ImageNet, QNLI, and SQuAD 1.1 across pruning budgets from 1% to 50% demonstrate competitive performance against a broad range of pruning methods. The paper also investigates label noise, cross-architecture transfer, and adaptation to unseen budgets. The approach is promising, although the generality of its theoretical justification and the accounting of its overall computational cost deserve further clarification.

Critical strengths
  1. The paper addresses an important limitation of dataset pruning by explicitly modeling how the preferred difficulty distribution changes with the pruning budget.
  2. DPS replaces repeated preference-parameter search with a single probing run and infers a flexible allocation from two complementary signals derived from training dynamics.
  3. The experiments cover five datasets across image classification, text classification, and question answering, with pruning budgets ranging from 1% to 50%.
  4. The results are particularly encouraging under aggressive pruning; for example, DPS achieves 18.02% accuracy on CIFAR-100 at a 1% budget, outperforming the strongest reported baseline by 2.06 percentage points.
  5. The ablation studies, label-noise experiments, and cross-architecture transfer evaluation provide useful evidence supporting the practical applicability of the proposed approach.
Critical weaknesses
  1. The theoretical analysis relies on a simplified linearized model with a specific shared-direction and example-specific feature structure, leaving the applicability of its conclusions to nonlinear deep networks insufficiently established.
  2. The excess-fitting signal compares errors from the probing subset and the full difficulty strata using different models and sample sets, which may introduce estimation noise and affect the reliability of the inferred allocation, particularly at small budgets.
  3. The claimed reduction in adaptation cost focuses on the additional probing runs, while computing the initial difficulty scores still requires full-data training; the overall computational savings relative to competing pruning pipelines are therefore not fully characterized.
  4. DPS does not consistently outperform all baselines across every dataset and budget, and several improvements are marginal, making the robustness of its advantage dependent on the evaluation setting.
  5. The theoretical justification for combining the two signals through a product depends on additional conditions, while the empirical behavior of the inferred preference may vary across datasets and difficulty-score distributions.
Questions for authors
  1. How well do the theoretical conclusions derived from the simplified linearized model explain the observed training dynamics of nonlinear deep networks across the evaluated datasets and architectures?
  2. How sensitive is the excess-fitting signal to the mismatch between the probing subset and the full difficulty strata, and can the authors quantify its estimation variance at small pruning budgets?
  3. Can the authors report a more comprehensive computational-cost comparison that includes difficulty-score computation, probing, subset construction, and any additional training required by competing methods?
  4. Under which datasets and pruning budgets does DPS fail to outperform existing methods, and can the authors identify the characteristics of these settings that limit the effectiveness of the inferred allocation?
  5. How robust is the product-based preference score when the two signals do not exhibit the expected opposing trends or when their profiles are nearly flat, and can the authors provide additional evidence beyond the current ablation study?
Reviewer_as9cOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DPS, which uses a single probing run and two training-dynamics signals to adapt dataset pruning to different budgets. Experiments across five datasets show improved performance over existing methods.

Critical strengths
  1. The paper addresses an important limitation of dataset pruning: the optimal balance between representative and difficult examples changes with the pruning budget.

  2. DPS offers an alternative to existing approaches that prescribe a difficulty preference and tune its parameters. Combining budget-conditioned difficulty with excess fitting provides an intuitive way to balance informativeness against overfitting.

  3. The experiments cover five datasets, multiple pruning budgets, different difficulty scores, and twelve baselines. The results suggest that DPS performs consistently well, particularly under aggressive pruning.

Critical weaknesses
  1. DPS combines two signals to decide which difficulty levels of data should be selected under different data budgets. Although the ablation studies show that both signals are useful, the results may depend on how the signals are calculated and combined. Further analysis is needed to show that this approach is robust and works well in different settings, rather than relying on these specific design choices.

  2. DPS only needs one probing run for each target budget, which helps reduce the tuning cost. However, the overall method also requires computing difficulty scores and using training checkpoints to compare model performance. It will be good to clarify the total computational cost and show when DPS is actually more efficient than other methods.

  3. The paper argues that using a uniform allocation for the probing run is a good choice under certain assumptions about unknown noise. However, it is unclear how this theoretical result helps improve pruning performance in practice. The paper should better explain how the theoretical analysis supports the effectiveness of DPS in the experiments.

Questions for authors
  1. Can the theoretical analysis more directly explain why the product of the two signals produces better allocations than alternatives, such as a weighted combination?

  2. How much of the total pruning cost is attributable to the probing run, difficulty-score computation, and checkpoint matching, and how does the complete pipeline compare with competing methods?

Comments on the paper

1 comment

Dear Reviewers,

The authors have finished uploading their responses to reviewers, and the discussion phase is now underway. If you haven't already, please carefully review the authors' responses to understand their perspectives. Please engage in thoughtful and constructive discussions with the authors, and update your review or rating if necessary.

The discussion period lasts until November 18th AoE, which means you have less than five days left to interact with the authors. Please start engaging well before the period ends, so that the authors will have sufficient time to respond to any follow-up questions/comments.

Your active participation and contribution to the ongoing discussion are highly encouraged. Thank you very much for your contribution to ICLR.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.