acceptodds
Desk rejected at ICLR 2027

Measuring Uneven Class Damage from Data Pruning: Four Pitfalls and a Control Ladder

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Score-based data pruning damages some classes more than others, and the standard way of measuring that damage cannot tell three explanations apart: the class imbalance a difficulty score induces, the choice of *which* examples survive inside each class, and the plain fact that a pruned model is less accurate. We document four measurement pitfalls that produce false positives here, each demonstrated on a conclusion of our own that we then withdrew: a disagreement statistic reported without its mechanical floor, which inflated a churn ratio to 2.08 when the floor accounts for essentially all of it; a baseline fitted through the very group under test; two non-pruning control families that disagree at matched accuracy; and selection-seed variance as large as training-seed variance, in a setting where 16.3% of a canonical Forgetting coreset is decided by score ties and only 94.3% of its indices are stable across seeds. Three of the four apply to any study that measures how much two models disagree. We then build a ladder of five increasingly strict controls, fixing budget, then per-class counts (Gini), then the hard/random mixing fraction q, then final accuracy, and run the same question up it. An aggregate hypothesis dies at the second rung: against a non-pruning control trained to the same accuracy, the difference is +0.0005, CI [-0.023,+0.023]. A class-level hypothesis survives all five on the setting where we found it, bracketed at [-1.26,-0.71] by two comparisons that fail in opposite directions, and persists under a second architecture. It does not generalize: borderline on Food-101, null under real human label noise, and sign-reversed on Tiny-ImageNet with an interval that excludes zero. Subsampling CIFAR-100's test set to Tiny-ImageNet's per-class count leaves the effect intact in 99% of draws, so the reversal is a boundary of the mechanism and not small-sample noise. We report the ladder, the failures, and the pitfalls as the portable contribution.

Primary area:
datasets and benchmarks
Keywords:
data pruning, coreset selection, class-level fairness, negative flip rate, prediction churn, experimental controls, negative results, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XimyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether score-based data pruning disproportionately harms some classes because it keeps different numbers of examples per class, chooses hard examples within each class, or simply produces a less accurate model. It audits common comparisons by progressively controlling these factors, including a hard-versus-random comparison with identical per-class sample counts and an approximate accuracy match obtained by shortening one model's training schedule. On CIFAR-100, the usual same-budget comparison against random pruning is badly confounded by class-count imbalance; after fixing those counts, selecting harder examples still concentrates damage in under-retained classes. However, this result only survives an architecture change on CIFAR-100: it is weak on Food-101, absent under noisy CIFAR-100 labels, and reverses on Tiny-ImageNet. The most credible contribution is therefore a practical audit protocol and a warning about controls and selection randomness, rather than a general finding that hard-example pruning harms classes.

Critical strengths
  1. A useful control design for separating sources of class-level damage. The qq-mixture holds per-class counts exactly fixed while varying the hard-versus-random composition within each class. Together with the approximate accuracy-matched control, it distinguishes "how many examples per class", "which examples", and overall accuracy more carefully than a same-budget Random baseline. This is the paper's clearest reusable contribution.

  2. Transparency about negative results and boundaries. The paper openly reports an abandoned aggregate hypothesis, a null result on CIFAR-100-N, a sign reversal on Tiny-ImageNet, and a borderline Food-101 result. Its reporting of score ties and selection-seed instability is also useful for reproducibility.

Critical weaknesses
  1. The portable contribution is useful but limited in methodological novelty. The paper explicitly positions the four pitfalls and the control ladder as its main portable contribution. The concrete count-matched qq-mixture and the quantification of tie-induced selection variability are useful, but several broader lessons—using an external baseline, validating matched controls, and replicating all relevant sources of randomness—are established experimental-design principles. The novelty therefore lies mainly in assembling and demonstrating these controls for data pruning, which limits the significance of the paper's central methodological claim.

  2. The discovery-setting effect contains an unresolved numerical discrepancy. In the Rung 5 paragraph of Section 5.3 (p. 6), the paper reports β~(q=1)=−3.460\tilde\beta(q{=}1)=-3.460 and β~(q=0,T=9000)=−2.208\tilde\beta(q{=}0,T{=}9000)=-2.208, but then states that their difference is Δβ~=−1.20\Delta\tilde\beta=-1.20; the displayed coefficients instead differ by −1.252-1.252. The value −1.252-1.252 also appears in the following "Close to parallel" paragraph as the fitted vertical gap at the high end of the accuracy overlap, while Table 3 repeats −1.20-1.20 as the headline CIFAR-100 result. The authors should clarify whether −1.20-1.20 is a separately pooled or bootstrapped estimate rather than the difference of the displayed coefficients, and state exactly how it is computed.

  3. The paper is unnecessarily difficult to follow. Rungs 1–2 are introduced mainly inside Figure 1, after which the prose in Section 4 begins directly with Rung 3; Section 5 then returns to Rung 1 and proceeds forward. The main paper also introduces local checkpoint terms such as "candidate anchor," "accuracy-matched anchor," and "screening/QC window." Although these terms are defined, later references often rely on the reader remembering both the rung number and the associated checkpoint construction. A more linear presentation that restates what each comparison holds fixed would make the otherwise straightforward experimental logic substantially easier to follow.

Questions for authors

The within-class effect changes substantially across CIFAR-100, Food-101, and Tiny-ImageNet. Since the utility of hard-example selection may depend on whether the selected training examples cover the difficult regions represented in the test distribution, have you estimated test-example difficulty using an independent model or ensemble and examined the effect separately on independently identified easy, medium-difficulty, and hard test examples? Such analyses could help distinguish a dataset-specific pruning mechanism from differences in difficulty composition and train–test alignment.

Reviewer_AkcZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the uneven impact of data pruning on classification performance across different classes, analyzes potential confounding factors in existing evaluation methods, and attempts to disentangle the effects of class sample size, intra-class sample selection methods, and changes in overall accuracy through a five-level control experimental framework (Control Ladder). The authors summarize four types of measurement issues and conduct experiments using pruning methods such as Forgetting, EL2N, and CCS. The results show that after controlling for accuracy, overall prediction differences may no longer be significant, while performance disparities caused by intra-class sample selection persist under certain experimental settings. However, this phenomenon varies inconsistently across different datasets. Overall, this work is primarily an empirical analysis of existing data pruning evaluation methods, rather than proposing new pruning algorithms or theoretical models.

Critical strengths
  1. The research question has practical significance.
    This paper addresses the issue of uneven impact on different classes during data pruning, emphasizing the need to consider confounding factors such as class distribution and overall model accuracy when evaluating results. This issue holds value for understanding the real-world implications of data pruning methods.

  2. The presentation of experimental results is relatively comprehensive.
    The authors not only report findings that support their research hypothesis but also present some non-significant or even contradictory results, while accounting for training randomness and random sample selection. These analyses help in understanding the instability of the experimental outcomes.

Critical weaknesses
  1. The paper lacks sufficient innovation, and its substantive methodological contributions are relatively limited.

This is my primary concern regarding the manuscript. Although it proposes four categories of measurement issues and a five-level control experimental framework, most of its content relies heavily on existing experimental design principles and statistical analysis methods—such as controlling for confounding variables, matching model accuracy, using random control groups, and analyzing variance in random seeds. The paper mainly reorganizes these established analytical approaches into a unified experimental workflow without introducing sufficiently novel theoretical analyses, evaluation mechanisms, or methodological designs. In particular, the authors have not adequately explained what unique methodological advantages the proposed Control Ladder offers over existing controlled variable experimental designs. Therefore, while the experimental workload appears substantial, the paper's substantive original research contribution remains relatively modest.

  1. The progressive logic of the Control Ladder is insufficiently rigorous, and the independent methodological value of the framework remains unclear.

The authors structure the five levels of experiments as a progressively increasing control condition framework; however, these experiments are not strictly nested. For example, the second-level experiment primarily focuses on overall accuracy matching, whereas the third-level experiment shifts to balancing class sample counts without maintaining the accuracy constraints from the second level. From this perspective, the framework resembles an organization and generalization of multiple distinct experimental control strategies rather than a coherent, theory-driven, stepwise evaluation method. Furthermore, although the paper demonstrates through multiple experiments that different control strategies can alter experimental conclusions, it has not sufficiently argued how this framework provides new analytical capabilities compared to conventional ablation studies or fairness control experiments.

  1. The generalizability of the core experimental findings is limited, and no compelling mechanistic explanation has been provided.

The paper observes on CIFAR-100 that within-class difficult sample selection exacerbates performance disparities across classes, but this finding is inconsistent across other datasets. Results on Food-101 lie near the threshold of statistical significance, no significant effect is observed on CIFAR-100-N, and Tiny-ImageNet even shows the opposite trend. While the authors truthfully report these results, they do not explain why the same sample selection strategy produces contradictory effects across different datasets. The authors rule out one possible explanation by conducting sub-sampling on the test set, but this does not suffice to establish a new mechanistic understanding. Consequently, the conclusions drawn in the paper remain largely confined to specific experimental settings, and their broader research implications remain unclear.

  1. The accuracy-matching strategy still introduces confounding factors, undermining the credibility of the core conclusions.

In the fifth-level experiment, the authors achieve accuracy matching by shortening the training steps of the random control model. However, this approach simultaneously introduces differences in training dynamics and optimization states. Although the two models achieve similar final accuracies, this does not necessarily imply equivalent class-level learning states. Thus, the observed class-level performance differences may partly stem from training dynamics rather than being entirely attributable to within-class sample selection. The authors attempt to address this by conducting experiments with varying training lengths, but the overlapping range of accuracies is narrow, which is insufficient to fully resolve this concern.

Questions for authors

Q1. The four types of measurement issues proposed in this paper largely involve common problems in existing experimental designs and statistical analyses. Could the authors further clarify what the most core and irreplaceable methodological innovation is, compared to current research on data pruning fairness and conventional control variable analysis?

Q2. Control Ladder is described as a framework that progressively adds control conditions, but the second- and third-level experiments do not satisfy a strict nested relationship in terms of control conditions. How do the authors justify this design? Compared to treating these experiments as independent control analyses, what additional methodological value does the five-level framework itself offer?

Q3. The core category-level effects in the paper show inconsistent or even contradictory trends across different datasets. Could the authors provide further mechanistic analyses—beyond merely reporting statistical differences—to explain this phenomenon?

Q4. The fifth-level experiment achieves accuracy matching by varying training steps. Could the authors further demonstrate that the observed differences primarily stem from sample selection strategies, rather than category learning dynamics at different training stages?

Reviewer_yhqQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies why removing training examples can reduce accuracy more for some classes than for others. It asks whether this happens because pruning retains different numbers of examples from each class, selects particular examples within each class, or simply because the resulting model is less accurate overall.

The authors identify four pitfalls in measuring pruning damage and propose five increasingly controlled comparisons. These include comparing pruning with a full-data model trained to similar accuracy, comparing selection methods that retain exactly the same number of examples per class, and gradually replacing randomly chosen examples with examples selected by a difficulty score. They also propose adjusted measures of prediction disagreement and class-level damage, and examine how results vary with randomness in both data selection and model training.

The experiments focus on CIFAR-100, with additional checks on Food-101, Tiny-ImageNet, and noisy training labels. The authors find that the apparent increase in overall prediction disagreement largely disappears when compared with a similarly accurate full-data model. However, selecting difficult examples rather than random examples still produces greater damage to classes with lower retention rates in the main CIFAR-100 setting and under the architecture change. This finding is weaker on Food-101, absent with noisy labels, and reversed on Tiny-ImageNet.

Critical strengths

The paper studies an important problem that average accuracy can hide. Pruning may harm some classes much more than others. Distinguishing between retaining fewer examples of a class and choosing different examples within that class helps clarify the sources of this harm.

The proposed evaluation framework uses several useful controls. In particular, preserving the exact number of examples in every class provides a stronger control than matching a single measure of class imbalance, since datasets with the same overall imbalance can retain very different numbers of examples from individual classes. Keeping the number of training updates fixed also helps separate the effects of data selection from those of reduced training. The evaluation considers several sources of uncertainty and variation. The authors examine randomness in data selection as well as model training, and vary the mixture of difficult and random examples, training duration, and network architecture. They also examine worst-class accuracy. Importantly, they report results that weaken, disappear, or reverse across datasets and noisy-label settings rather than presenting the findings as universal. This helps clarify the limits of their conclusions.

Critical weaknesses

The paper is difficult to follow and needs substantial restructuring. The introduction points to Section 6 for pitfalls that motivate Sections 4-5, and the control ladder is not explained in a straightforward order. Equations are unnumbered, definitions are scattered, and phrases such as "we report the autopsies" add rhetoric without clarifying the analysis. Additionally, the novelty claims need more careful positioning. DRoP [1] already includes random selection that respects class-specific quotas, and earlier work studies accuracy-aware prediction agreement. The paper should explain more clearly what its combined evaluation framework adds beyond these existing ideas.

The interpretation of the proposed metrics is not sufficiently precise. The disagreement reference is called a "floor" in the text and a "ceiling" in Figure 2, although it appears to be an expectation under particular assumptions rather than a universal bound. Moreover, its change does not, by itself, explain the reported disagreement ratio of 2.08. Similarly, the proposed normalization does not generally remove the influence of accuracy: it handles a particular pattern in which all class-level negative-flip rates increase by the same proportion. Since accuracy changes need not follow this pattern, the adjusted coefficient has a narrower interpretation than the paper suggests. Moreover, the quantitative evidence needs clearer reporting and stronger statistical support. Different sections report different values for apparently related effects without clearly identifying the seeds, averaging procedures, or comparisons involved. The test-subsampling analysis also appears to use a different comparison from the main accuracy-matched result, weakening its interpretation. In addition, key comparisons use only two or three independent data selections, limiting confidence in the estimated variability. Finally, an interval containing zero shows that a difference was not detected; it does not establish that the effect is absent.

[1] Vysogorets, A. M., Ahuja, K., & Kempe, J. (2025). DRoP: Distributionally Robust Data Pruning. The Thirteenth International Conference on Learning Representations.

Questions for authors
  1. What does the proposed disagreement reference represent? It is described as a "floor" in Section 3 and a "ceiling" in Figure 2, although it appears to be neither a general lower nor upper bound. Can you clarify its assumptions and explain how it accounts for the reported disagreement ratio of 2.08?

  2. How do you separate the effect of data selection from the effect of training duration? Matching accuracy requires training the models for different numbers of updates. What assumptions justify interpreting this comparison as evidence about selection, and why should the same-schedule and accuracy-matched comparisons bound the true effect?

  3. Can you reconcile the different reported values of the main effect? Please identify the comparisons, seeds, and averaging procedures behind the values in Section 5.3, Tables 2 and 4, and Appendix A. In particular, does the test-subsampling experiment evaluate the same accuracy-matched comparison used for the main conclusion?

1 comment

Dear authors,

Thank you for the constructive response. The narrower scope addresses several of my concerns.

Two issues remain unresolved:

  • The estimates in Section 5.3, Tables 2 and 4, and Appendix A remain unreconciled.
  • The presentation concerns remain insufficiently addressed, and the contribution beyond existing quota-based controls is still unclear.

Although the response improves my overall assessment, the main numerical results remain unverified, and the evidence does not yet establish the narrower contribution clearly enough to support acceptance. Addressing these issues, alongside the changes already proposed, would amount to a substantial revision whose soundness and significance would benefit from a new round of review. I therefore retain my current recommendation.

Comments on the paper

1 comment

Dear Reviewers, please take a moment to read the author responses and engage in the discussion. In particular, please comment on whether your questions and concerns have been adequately addressed and follow up on any points that remain unclear. Thank you! AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.