acceptodds
Withdrawn from ICLR 2027

Are Frequency-Balanced Datasets Optimal for Equitable Class Performance?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Resampling is a widely used approach for mitigating class-wise performance disparities in frequency-imbalanced datasets, typically by reallocating training data according to class frequency. However, this framing treats representation as the primary basis for deciding how much data each class should receive, while classes may also differ substantially in learning difficulty. Hardness-aware resampling and reweighting have shown that allocating more training resources to harder classes can improve class-wise fairness, but existing evidence comes predominantly from frequency-imbalanced settings, where the effects of correcting representation and accounting for hardness are difficult to disentangle It therefore remains **unclear whether hardness provides a useful basis for data allocation in frequency *balanced* settings**. We investigate this question by deliberately allocating more training data to harder classes. We find that this reallocation reduces class-wise recall disparities while maintaining comparable overall performance. We further show that the effectiveness of oversampling depends on the quality of the added samples, with hardness emerging as a strong estimate for downstream task utility. These results demonstrate that **resampling can be useful even when there is no representation imbalance to correct**, and motivate greater attention to class-wise fairness in frequency-balanced settings.

Primary area:
interpretability and explainable AI
Keywords:
data imbalance, hardness imbalance, class bias, hardness, resampling, diffusion models, fairness-aware learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1h6QOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Resampling approaches for mitigating class-wise performance disparities predominantly focus on reallocating training data according to class frequency, however, this setting ignores that classes also differ in learning difficulty. Some following works validate that class hardness provide additional basis for allocating training resources across classes, but these evidence are still mostly under the frequency-imbalanced setting, which can not disentangle the contribution from correcting the representation deficit and hardness-guided resources allocation. This work is therefore motivated to test the hardness-based allocation under the frequency-balanced setting, which attempts to disentangle the above two parts. Two complementary case studies illustrate that hardness-guided resampling strategy could reduce class-wise recall gap among different classes, which shows that frequency imbalance is not the only factor that results in class-wise performance disparity.

Critical strengths
  1. The problem itself is meaningful, this paper highlights that the inherent learning hardness of a class itself can also lead to issues of class-wise fairness beyond class frequency.
  2. The comparison among three oversampling strategies (rEDM, aEDM, and hEDM) supports the hardness of oversampled data determines its utility for improving class-wise equity.
Critical weaknesses
  1. The main investigation in this paper is only two case studies under the frequency-balanced setting, which is too limited to convince the readers. More importantly, this paper is positioned as expanding the hardness-base resampling effectiveness evidence from frequency-imbalanced dataset to frequency-balanced one. As a phenomenon investigation paper, I can not find any additional finding or conclusion from it given similar existing work [1], a comprehensive novelty clarification is needed during the rebuttal.
  2. In line 74, the author claimed that "This creates an important ambiguity: when ... Testing hardness-based allocation in a frequency-balanced setting addresses this ambiguity". However, it is hard to understand why testing under a frequency-balanced setting could addresses the ambiguity. The hardness-based reallocation on training dataset also changes the number of samples of the hard class, which may result in the observed improvement of class-wise equity.

[1] "Reducing Class Bias In Data-Balanced Datasets Through Hardness-Based Resampling", Pawel Pukowski, Venet Osmani. https://arxiv.org/abs/2504.07031

Questions for authors
  1. There are two repeated sentences in line 87-89.
  2. Is it reasonable to use recall gap as a representative metric for class-wise equity? Does the reduction in the recall gap come at the cost of increasing the gap in other class-wise equity metrics?
  3. The whole section 3 and 4 are centered on the two case studies, is there any deep analysis to help understand why hardness-based reallocation could be effective for reducing class-wise recall gap.
  4. How to define the "hardness" and estimate it?
Reviewer_3nzkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper discusses resampling of training data. In particular, rather than focusing on traditional considerations of frequency of occurrence, it approaches the topic from the perspective of training difficulty. Using benchmark results, it provides insights into resampling in relation to training difficulty.

Critical strengths

In the discussion regarding data imbalance, this offers a new perspective: unlike simply balancing the distribution of occurrences, it is necessary to consider the issue in terms of training difficulty.

Critical weaknesses

There are two main concerns regarding this paper.

The first is novelty. There have been numerous previous discussions on resampling, and it is believed that discussions regarding learning difficulty are already included in those. [1] dynamically adjusts sampling based on difficulty, while [2] performs dynamic resampling to converge toward a target learning accuracy. Although both assume imbalanced data, this is primarily because such data is more susceptible to the effects of resampling; the methods are still applicable even when the data is balanced from the outset. While the debate over whether resampling is necessary even when data is balanced is a new one, it is not simply acceptable to claim that insights on this matter are absent from these previous studies.

Second, this paper does not propose a new method but is merely a foundational experiment designed to gather insights on difficulty-dependent resampling. While I do not dismiss such research, from a broader perspective—and given the existence of similar studies—it is questionable whether the insights provided by this foundational experiment meet the acceptance standards of this top-tier conference. Furthermore, the insights obtained are based on only two case studies, and it is unclear whether they are universally applicable.

[1] - I. Naglik and M. Lango, GMMSampling: a new model-based, data difficulty-driven resampling method for multi-class imbalanced data, Machine Learning Vol.113, 5183–5202 (2024)

[2] S. Ramasubramanian et al., Selective Mixup Fine-Tuning for Optimizing Non-Decomposable Objectives, ICLR2024

Questions for authors

Among the points presented in this paper, are there any issues or perspectives—other than the fact that the original data is imbalanced—that have not been pointed out in papers such as [1] and [2]?

Reviewer_AW8zOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates from the data-centric perspective, how to improve class performance disparity. They use hardness based resampling to train the network and hardness is defined by area under the margin. They found that hardness based resampling can reduce class-wise recall disparities, and EDM-generated hard sample is more effective. All experiments are conduced with ResNet-18 on CIFAR-10 and CIFAR-100.

Critical strengths
  1. I like the idea that from a data-centric perspective, we can address the current issues of machine learning.

  2. The overall experiment design is rigorous and systematic.

  3. The conclusion I personally think is valid.

Critical weaknesses
  1. The definition of hard or sample is inaccurate, as we can see from equation (1), it seems to tell which sample is harder to fit and which sample is easier to fit, like support vectors. It does not reflect its intrinsic difficulty and generalization utility. Focusing on those hard samples can be overfitting the network.

  2. An outlier can be a hard sample, if the data augmented contain a lot of outliers, it also can hurt the performance.

  3. The distribution shift between test and training sample is not taken into account. And such distribution shift can be very common. For example, spurious correlation in the training set but not in the test set. Would the same conclusion be valid?

  4. Given the current conclusion, from the data perspective to encourage class-balanced performance, we need to train the entire model and score on the training samples. It can be very expensive on the larger dataset.

  5. Hard example mining has been studied in imbalanced learning. E.g. class rectification for imbalanced deep learning shows hard sample mining for class imbalance. I would like to see a comparison with representative hard example mining approaches and class level hardness resampling.

Questions for authors
  1. For computing AUM, because equation (1) is average on all training step, I am curious those AUM are computed based on training samples, or validation samples.

  2. Why don't we directly use class accuracy to reflect class-fairness instead of recall.

  3. Would different training objectives change the conclusion, like using supervised contrastive learning, self-supervised learning with linear prob, or contrastive learning with hard negative samples (https://arxiv.org/pdf/2010.04592)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.